Nov 02 21:55:03.894347 ip-10-0-136-9 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 21:55:03.894356 ip-10-0-136-9 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 21:55:03.894362 ip-10-0-136-9 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 21:55:03.894594 ip-10-0-136-9 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 21:55:13.898279 ip-10-0-136-9 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 21:55:13.898296 ip-10-0-136-9 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 558fdbccee6f4202994df73b85571922 -- Nov 02 21:57:10.567326 ip-10-0-136-9 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:57:10.945864 ip-10-0-136-9 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:57:10.945864 ip-10-0-136-9 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:57:10.945864 ip-10-0-136-9 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:57:10.945864 ip-10-0-136-9 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:57:10.951949 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951869 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:10.951949 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951948 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:10.951949 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951952 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951955 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951959 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951962 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951964 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951967 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951970 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951972 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951975 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951978 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951981 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951983 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951986 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951989 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951992 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951995 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.951997 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952000 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952002 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:10.952058 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952011 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952014 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952016 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952019 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952022 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952024 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952027 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952029 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952032 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952037 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952055 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952059 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952062 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952065 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952068 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952072 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952074 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952077 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952079 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:10.952496 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952082 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952085 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952087 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952090 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952093 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952096 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952099 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952101 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952104 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952106 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952108 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952111 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952113 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952116 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952118 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952122 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952124 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952127 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952130 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:10.952941 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952132 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952135 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952138 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952140 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952143 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952146 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952149 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952153 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952157 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952161 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952163 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952166 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952169 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952172 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952174 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952177 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952180 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952183 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952185 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:10.953403 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952188 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952191 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952194 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952196 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952199 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952202 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952204 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952207 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952209 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952213 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952222 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952225 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952228 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952230 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952233 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952235 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952238 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952241 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952244 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952246 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:10.953847 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952249 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952251 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952254 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952256 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952259 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952261 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952264 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.952267 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953920 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953929 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953933 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953936 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953938 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953942 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953945 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953948 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953951 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953954 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953956 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:10.954364 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953959 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953962 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953965 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953967 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953970 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953972 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953975 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953977 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953980 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953982 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953985 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953988 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953990 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953993 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953996 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.953999 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954003 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954006 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954008 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954011 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:10.954821 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954014 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954017 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954019 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954021 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954024 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954027 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954029 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954032 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954035 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954038 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954052 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954055 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954058 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954061 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954064 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954067 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954070 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954072 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954075 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:10.955339 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954077 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954080 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954083 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954085 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954088 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954091 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954094 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954098 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954102 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954105 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954108 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954111 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954114 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954118 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954121 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954124 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954126 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954129 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954132 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954134 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:10.955780 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954137 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954140 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954143 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954145 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954148 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954150 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954153 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954156 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954158 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954161 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954164 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954166 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954169 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954172 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954174 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954184 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954187 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954189 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954192 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954195 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:10.956256 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954198 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954202 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954204 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954207 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954210 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954212 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954216 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954221 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954224 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954227 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954230 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954233 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954235 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954238 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954241 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954244 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954743 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954748 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954751 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:10.956751 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954754 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954757 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954759 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954763 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954766 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954769 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954771 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954774 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954776 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954779 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954781 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954785 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954787 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954790 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954793 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954795 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954798 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954800 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954803 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954806 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:10.957209 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954808 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954811 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954813 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954816 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954818 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954821 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954824 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954827 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954830 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954833 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954836 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954838 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954849 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954852 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954855 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954858 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954861 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954864 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954868 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:10.957688 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954872 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954875 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954877 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954880 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954883 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954885 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954888 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954891 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954893 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954896 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954899 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954901 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954904 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954906 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954909 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954912 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954914 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954917 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954919 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:10.958150 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954922 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954924 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954929 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954931 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954934 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954937 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954939 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954944 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954947 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954949 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954952 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954954 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954957 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954959 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954962 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954964 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954967 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954969 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954972 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954975 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:10.958628 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954977 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954981 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954985 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954988 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954991 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954993 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954996 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.954998 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955001 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955003 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955006 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955008 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955011 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955013 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955016 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955020 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955023 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955026 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955028 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:10.959167 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955033 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955035 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955038 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955040 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955056 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.955059 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956141 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956152 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956159 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956164 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956168 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956172 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956177 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956181 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956185 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956188 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956193 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956196 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956199 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956203 2578 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956206 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956209 2578 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956212 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:57:10.959615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956215 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956220 2578 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956223 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956226 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956229 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956233 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956237 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956241 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956244 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956250 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956253 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956256 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956260 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956263 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956267 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956271 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956274 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956277 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956280 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956284 2578 flags.go:64] FLAG: --enable-server="true" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956287 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956291 2578 flags.go:64] FLAG: --event-burst="100" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956294 2578 flags.go:64] FLAG: --event-qps="50" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956297 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:57:10.960174 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956300 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956304 2578 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956308 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956311 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956315 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956318 2578 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956321 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956324 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956327 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956330 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956333 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956336 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956339 2578 flags.go:64] FLAG: --feature-gates="" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956343 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956346 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956350 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956354 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956357 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956360 2578 flags.go:64] FLAG: --help="false" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956363 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956367 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956370 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956373 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:57:10.960762 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956376 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956379 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956382 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956385 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956388 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956391 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956394 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956397 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956400 2578 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956403 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956406 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956409 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956412 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956415 2578 flags.go:64] FLAG: --lock-file="" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956418 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956421 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956424 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956430 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956432 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956435 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956439 2578 flags.go:64] FLAG: --logging-format="text" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956442 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956445 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956448 2578 flags.go:64] FLAG: --manifest-url="" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956451 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:57:10.961324 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956462 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956465 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956473 2578 flags.go:64] FLAG: --max-pods="110" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956477 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956480 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956483 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956486 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956489 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956492 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956495 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956503 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956506 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956509 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956512 2578 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956515 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956518 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956521 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956524 2578 flags.go:64] FLAG: --port="10250" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956527 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956530 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-043b2de84a6d983eb" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956533 2578 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956536 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956539 2578 flags.go:64] FLAG: --register-node="true" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956542 2578 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956545 2578 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956548 2578 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:57:10.962001 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956551 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956554 2578 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956558 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956561 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956564 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956567 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956570 2578 flags.go:64] FLAG: --runonce="false" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956573 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956576 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956579 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956583 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956585 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956589 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956592 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956595 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956598 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956601 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956604 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956606 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956610 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956613 2578 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956615 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956621 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956624 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956627 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:57:10.962673 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956631 2578 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956634 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956637 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956640 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956643 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956646 2578 flags.go:64] FLAG: --v="2" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956650 2578 flags.go:64] FLAG: --version="false" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956654 2578 flags.go:64] FLAG: --vmodule="" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956660 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.956663 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956814 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956819 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956822 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956825 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956827 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956831 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956833 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956836 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956839 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956841 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956844 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:10.963610 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956846 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956850 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956854 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956858 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956862 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956865 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956868 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956871 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956876 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956879 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956882 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956886 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956888 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956891 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956894 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956897 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956899 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956902 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956904 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:10.964600 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956911 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956914 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956916 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956919 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956922 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956925 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956928 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956930 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956934 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956937 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956939 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956942 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956944 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956947 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956950 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956952 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956955 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956957 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956960 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956962 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:10.965416 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956965 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956969 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956972 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956975 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956977 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956980 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956983 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956985 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956988 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956990 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956993 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.956995 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957000 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957002 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957005 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957008 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957010 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957013 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957015 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957018 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:10.966275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957020 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957024 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957026 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957029 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957031 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957034 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957037 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957039 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957054 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957057 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957060 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957062 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957065 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957069 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957071 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957074 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957077 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957079 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957082 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957085 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:10.967166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957087 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957090 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957093 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957095 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957099 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957101 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957104 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957107 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957109 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957112 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957114 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957117 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957120 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957122 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957125 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957128 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:10.968062 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.957156 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957193 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957198 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957203 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957206 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957209 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957212 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957215 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957218 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957221 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957225 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957228 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957231 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957233 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957236 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957239 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957241 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957244 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957246 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:10.968557 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957249 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957253 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957256 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957258 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957261 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957263 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957266 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957269 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957271 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957274 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957276 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957279 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957288 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957291 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957293 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957296 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957298 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957301 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957303 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:10.969233 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957306 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957309 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957312 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957314 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957318 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957321 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957323 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957326 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957329 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957331 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957334 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957337 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957339 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957342 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957346 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957349 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957351 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957354 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957357 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957359 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:10.970020 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957362 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957364 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957367 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957369 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957372 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957375 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957378 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957381 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957384 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957387 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957390 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957392 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957395 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957397 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957400 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957402 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957406 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957408 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957411 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957414 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:10.970708 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957416 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957419 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957421 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957424 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957427 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957429 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957433 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957435 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957438 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957441 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957443 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957446 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957448 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957451 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957453 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957456 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957458 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957461 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957464 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957466 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:10.971342 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957469 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957471 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957474 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957477 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957479 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957482 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957484 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957487 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.957492 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.965951 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.965977 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.965995 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.966006 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966115 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966124 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966130 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966134 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966139 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966143 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:10.971869 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966148 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966152 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966156 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966161 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966165 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966169 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966173 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966177 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966182 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966186 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966190 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966194 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966199 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966203 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966208 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966212 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966217 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966222 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966226 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966230 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:10.972379 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966234 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966243 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966257 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966262 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966266 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966271 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966275 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966279 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966283 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966287 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966291 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966294 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966299 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966303 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966307 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966311 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966316 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966320 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966325 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:10.972882 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966329 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966333 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966338 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966343 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966347 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966351 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966355 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966360 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966364 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966368 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966372 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966377 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966381 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966385 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966389 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966395 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966407 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966412 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966417 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:10.973456 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966421 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966425 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966430 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966434 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966438 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966442 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966446 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966451 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966455 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966459 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966463 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966468 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966472 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966476 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966481 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966485 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966489 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966493 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966498 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966502 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:10.973904 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966506 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966510 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966514 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966518 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966522 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966526 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966530 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966545 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966550 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966555 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966567 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966572 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966576 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966581 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966588 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966595 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966602 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966607 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966611 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:10.974448 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966615 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966620 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.966662 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966949 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966958 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966963 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966968 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966972 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966976 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966980 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966984 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966989 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966993 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.966998 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967002 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967006 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:10.974895 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967010 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967015 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967019 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967023 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967027 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967031 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967036 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967040 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967069 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967073 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967078 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967082 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967086 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967090 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967095 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967099 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967103 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967108 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967112 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967116 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:10.975321 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967120 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967124 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967128 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967133 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967137 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967141 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967145 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967149 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967154 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967158 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967162 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967166 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967170 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967174 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967178 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967182 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967186 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967192 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967199 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:10.975791 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967204 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967209 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967222 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967226 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967230 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967235 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967239 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967243 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967248 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967253 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967257 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967261 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967266 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967270 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967274 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967278 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967282 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967286 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967290 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:10.976565 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967295 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967299 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967303 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967308 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967319 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967324 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967328 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967332 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967336 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967341 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967348 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967353 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967358 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967363 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967368 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967372 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967384 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967389 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967393 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:10.977340 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967397 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967401 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967406 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967410 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967414 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967418 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967423 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967427 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967431 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967435 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967439 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967443 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967447 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967452 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967456 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967461 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:10.978077 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.967501 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967563 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967570 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967574 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967579 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967583 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967587 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967591 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967596 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967600 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967604 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967609 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967613 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967617 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967628 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967632 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967637 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967641 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967645 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967649 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:10.978684 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967653 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967661 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967665 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967669 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967674 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967678 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967683 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967687 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967691 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967695 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967699 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967706 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967713 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967718 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967723 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967727 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967732 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967736 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967740 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:10.979240 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967744 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967749 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967753 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967757 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967761 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967766 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967770 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967774 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967785 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967790 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967794 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967798 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967802 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967806 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967812 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967817 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967821 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967825 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967829 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967833 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:10.979686 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967837 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967841 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967845 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967851 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967857 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967861 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967865 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967869 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967874 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967878 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967882 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967886 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967890 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967895 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967899 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967915 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967919 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967923 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967928 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967932 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:10.980166 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967936 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967947 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967952 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967956 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967960 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967964 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967971 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967976 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967980 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967985 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967989 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967993 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.967998 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968002 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968006 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968010 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968015 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968019 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968023 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:10.980722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968027 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968031 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968035 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968039 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968057 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968062 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968066 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968069 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:10.968073 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.968159 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.971217 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.972338 2578 server.go:1014] "Starting client certificate rotation" Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.972436 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.972478 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:57:10.981290 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.976627 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:57:10.982729 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.982712 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:57:10.993905 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.993890 2578 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:57:10.997653 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.997638 2578 log.go:25] "Validated CRI v1 image API" Nov 02 21:57:10.999618 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:10.999588 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:57:11.003514 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.003490 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:57:11.004157 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.004134 2578 fs.go:135] Filesystem UUIDs: map[3f301e84-194d-4e77-919f-9a6b3385d3f0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 da48cd05-0025-45b6-b526-edb7a6de925a:/dev/nvme0n1p3] Nov 02 21:57:11.004208 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.004157 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 21:57:11.009320 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.009201 2578 manager.go:217] Machine: {Timestamp:2026-11-02 21:57:11.007856634 +0000 UTC m=+0.335018822 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099913 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23a626a4d5885ce155f7e72b857a90 SystemUUID:ec23a626-a4d5-885c-e155-f7e72b857a90 BootID:558fdbcc-ee6f-4202-994d-f73b85571922 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:d9:b2:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:d9:b2:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:0b:7f:89:65:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:57:11.009320 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.009318 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:57:11.009449 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.009436 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:57:11.010698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.010680 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:57:11.010835 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.010700 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-9.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:57:11.010882 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.010837 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:57:11.010882 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.010851 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:57:11.010939 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.010890 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:57:11.010939 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.010909 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:57:11.012220 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.012209 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:57:11.016909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.016897 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:57:11.017022 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.017013 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:57:11.017063 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.017056 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:57:11.017099 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.017066 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:57:11.017099 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.017076 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:57:11.017099 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.017085 2578 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:57:11.017099 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.017093 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:57:11.017449 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.017434 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:57:11.019956 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.019941 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:57:11.026990 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.026973 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:57:11.028018 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.028005 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:57:11.028091 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.028036 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:57:11.028272 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.028261 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:57:11.028311 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.028278 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:57:11.028311 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.028286 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:57:11.028311 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.028295 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:57:11.028311 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.028305 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:57:11.028407 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.028313 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:57:11.029007 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.028994 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:57:11.029007 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.029004 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:57:11.029121 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.029016 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:57:11.029121 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.029023 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:57:11.029121 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.029036 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:57:11.029121 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.029060 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:57:11.029571 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.029554 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mnjjv" Nov 02 21:57:11.029847 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.029837 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:57:11.029847 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.029847 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:57:11.035819 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.035797 2578 server.go:1263] "Started kubelet" Nov 02 21:57:11.035923 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.035845 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:57:11.036251 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.035895 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:57:11.036316 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.036288 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:57:11.036565 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.036540 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mnjjv" Nov 02 21:57:11.036682 ip-10-0-136-9 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:57:11.039090 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.039072 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:57:11.042066 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.040431 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:57:11.042066 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.041447 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-9.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 21:57:11.042283 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.042080 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-9.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 21:57:11.042283 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.042148 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 21:57:11.042283 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.042159 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 21:57:11.042283 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.041147 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-9.us-east-2.compute.internal.18e45933e5e90bc2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-9.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-9.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 21:57:11.035759554 +0000 UTC m=+0.362921743,LastTimestamp:2026-11-02 21:57:11.035759554 +0000 UTC m=+0.362921743,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-9.us-east-2.compute.internal,}" Nov 02 21:57:11.045820 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.045805 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:57:11.045870 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.045818 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:57:11.045927 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.045897 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.046587 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046569 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:57:11.046587 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046572 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:57:11.046587 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046595 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:57:11.046753 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046609 2578 factory.go:55] Registering systemd factory Nov 02 21:57:11.046753 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046623 2578 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:57:11.046753 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046638 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:57:11.046753 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046648 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:57:11.046877 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046826 2578 factory.go:153] Registering CRI-O factory Nov 02 21:57:11.046877 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046836 2578 factory.go:223] Registration of the crio container factory successfully Nov 02 21:57:11.046933 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046883 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:57:11.046933 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046897 2578 factory.go:103] Registering Raw factory Nov 02 21:57:11.046933 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.046924 2578 manager.go:1199] Started watching for new ooms in manager Nov 02 21:57:11.048011 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.047990 2578 manager.go:319] Starting recovery of all containers Nov 02 21:57:11.048118 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.048100 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-9.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 21:57:11.058805 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.058778 2578 manager.go:324] Recovery completed Nov 02 21:57:11.061688 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.061664 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 21:57:11.061748 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.061671 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:57:11.064226 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.064204 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:11.065198 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.065183 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:57:11.067000 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.066979 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:57:11.067090 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.067008 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:57:11.067090 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.067019 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:57:11.067528 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.067515 2578 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:57:11.067528 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.067525 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:57:11.067615 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.067540 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:57:11.069119 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.069104 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:57:11.069766 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.069753 2578 policy_none.go:50] "Start" Nov 02 21:57:11.069821 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.069782 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:57:11.069821 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.069796 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:57:11.069821 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.069806 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:57:11.069937 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.069833 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:57:11.069937 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.069844 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:57:11.071925 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.071913 2578 policy_none.go:44] "Start" Nov 02 21:57:11.104430 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.104404 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:57:11.106778 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.106762 2578 manager.go:341] "Starting Device Plugin manager" Nov 02 21:57:11.106844 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.106796 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:57:11.106844 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.106807 2578 server.go:87] "Starting device plugin registration server" Nov 02 21:57:11.107107 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.107094 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:57:11.107149 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.107109 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:57:11.107204 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.107188 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:57:11.107288 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.107269 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:57:11.107336 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.107290 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:57:11.107892 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.107875 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:57:11.107980 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.107923 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.188410 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.188384 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:57:11.188410 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.188413 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:57:11.188588 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.188438 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:57:11.188588 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.188477 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 21:57:11.191754 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.191737 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:11.208208 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.208140 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:57:11.209023 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.209005 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:57:11.209145 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.209036 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:57:11.209145 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.209067 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:57:11.209145 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.209106 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.219150 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.219135 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.219231 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.219157 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-9.us-east-2.compute.internal\": node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.234545 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.234527 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.289130 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.289102 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal"] Nov 02 21:57:11.289246 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.289160 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:57:11.290522 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.290508 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:57:11.290600 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.290536 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:57:11.290600 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.290547 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:57:11.292591 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.292577 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:57:11.292743 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.292727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.292804 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.292766 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:57:11.293384 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.293365 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:57:11.293460 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.293371 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:57:11.293460 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.293412 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:57:11.293460 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.293423 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:57:11.293460 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.293395 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:57:11.293595 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.293469 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:57:11.295764 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.295748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.295854 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.295772 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:57:11.296459 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.296442 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:57:11.296546 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.296470 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:57:11.296546 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.296484 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:57:11.327377 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.327359 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" node="ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.330438 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.330422 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" node="ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.334762 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.334748 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.348031 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.348006 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.348131 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.348039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.348131 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.348076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad3b40d52cd903e26b09603b8260563a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"ad3b40d52cd903e26b09603b8260563a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.435099 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.435076 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.448421 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.448400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.448486 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.448428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.448486 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.448447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad3b40d52cd903e26b09603b8260563a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"ad3b40d52cd903e26b09603b8260563a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.448557 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.448493 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.448557 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.448503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.448651 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.448555 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad3b40d52cd903e26b09603b8260563a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"ad3b40d52cd903e26b09603b8260563a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.535832 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.535807 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.631338 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.631297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.633443 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.633427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:11.636306 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.636164 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.737125 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.736935 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.837494 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.837425 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.869542 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.869515 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:11.937736 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:11.937709 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:11.972167 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.972143 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 21:57:11.972556 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.972289 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:57:11.972556 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.972316 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:57:11.972556 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:11.972315 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 21:57:12.038689 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:12.038657 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:12.045442 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:12.045391 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:52:12 +0000 UTC" deadline="2028-05-06 04:16:01.022081401 +0000 UTC" Nov 02 21:57:12.045442 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:12.045441 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13206h18m48.976643854s" Nov 02 21:57:13.477848 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:13.477789 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:13.578530 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:13.578499 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:13.634179 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:13.634139 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3b40d52cd903e26b09603b8260563a.slice/crio-f172b60e0b85874cb3ddda149bd16d5149c21eaef57032b0da728586f2b97831 WatchSource:0}: Error finding container f172b60e0b85874cb3ddda149bd16d5149c21eaef57032b0da728586f2b97831: Status 404 returned error can't find the container with id f172b60e0b85874cb3ddda149bd16d5149c21eaef57032b0da728586f2b97831 Nov 02 21:57:13.634468 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:13.634447 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d5d23fd92916dc8b8ecfbd0ec29fad.slice/crio-4f82fb253d3e5ce43f20e9f3437d710ed427d55dde183183327a8196762bbdfe WatchSource:0}: Error finding container 4f82fb253d3e5ce43f20e9f3437d710ed427d55dde183183327a8196762bbdfe: Status 404 returned error can't find the container with id 4f82fb253d3e5ce43f20e9f3437d710ed427d55dde183183327a8196762bbdfe Nov 02 21:57:13.639687 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:13.639659 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:57:13.679558 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:13.679535 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:13.780110 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:13.780034 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:13.880846 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:13.880820 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:13.900711 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:13.900687 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:13.984879 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:13.984853 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:13.996115 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:13.996099 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:57:13.996918 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:13.996905 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:14.005002 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.004981 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:57:14.268111 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.268080 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:14.356747 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.356712 2578 apiserver.go:52] "Watching apiserver" Nov 02 21:57:14.367970 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.367944 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:57:14.368640 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.368611 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-6jc58","openshift-multus/network-metrics-daemon-4l48q","openshift-multus/multus-dg6dc","kube-system/konnectivity-agent-zxdmr","openshift-cluster-node-tuning-operator/tuned-rszqx","openshift-dns/node-resolver-kclzj","openshift-image-registry/node-ca-jg7pl","kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-pft59","openshift-network-operator/iptables-alerter-lqhnn","openshift-ovn-kubernetes/ovnkube-node-pgrkb","kube-system/global-pull-secret-syncer-qkl72","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst"] Nov 02 21:57:14.371077 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.371053 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.373246 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.373220 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:14.373329 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.373306 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:14.375336 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.375315 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:57:14.375556 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.375541 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:57:14.375737 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.375706 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dx8bh\"" Nov 02 21:57:14.375820 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.375751 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:57:14.375820 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.375805 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.377751 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.377732 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:14.380127 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.380109 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.382712 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.382180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.384404 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.384311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.386110 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.386094 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 21:57:14.387068 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.387051 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.390049 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.389861 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:14.390049 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.389913 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:14.392522 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.392506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:14.392611 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.392568 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:14.394643 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.394624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.395558 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-conf-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.395558 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/735421c1-678c-4cc4-a07e-57ffb0e9ca29-hosts-file\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.395558 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.395778 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.395778 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.395778 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-socket-dir-parent\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.395778 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-netns\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.395778 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/735421c1-678c-4cc4-a07e-57ffb0e9ca29-tmp-dir\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.395778 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/527b2ac7-739e-4f53-b553-474cad1eb712-host\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.395778 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10158047-423e-4241-b46d-6ad29dece441-agent-certs\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-socket-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-dbus\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-var-lib-kubelet\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395909 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/527b2ac7-739e-4f53-b553-474cad1eb712-serviceca\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-kubernetes\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.395987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbh7g\" (UniqueName: \"kubernetes.io/projected/92283fa1-fe9a-4287-a313-176acb992dc9-kube-api-access-dbh7g\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/527b2ac7-739e-4f53-b553-474cad1eb712-kube-api-access-6rqjx\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtsdx\" (UniqueName: \"kubernetes.io/projected/735421c1-678c-4cc4-a07e-57ffb0e9ca29-kube-api-access-wtsdx\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-k8s-cni-cncf-io\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-conf\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-etc-tuned\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-multus-daemon-config\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.396203 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-etc-kubernetes\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqfz5\" (UniqueName: \"kubernetes.io/projected/79214604-184d-4eb5-b63e-d2a30209835c-kube-api-access-mqfz5\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396174 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-sys\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396194 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j57h\" (UniqueName: \"kubernetes.io/projected/53101ce6-458d-4b8f-ac21-36679d929ba0-kube-api-access-9j57h\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-run\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-sys-fs\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-device-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396316 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfc7z\" (UniqueName: \"kubernetes.io/projected/67af9038-04a3-4574-96fd-23a3bc52d944-kube-api-access-vfc7z\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-cnibin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-kubelet-config\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysconfig\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396404 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-tmp\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-kubelet\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396438 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-lib-modules\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.396989 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-multus-certs\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396506 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-host\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396526 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10158047-423e-4241-b46d-6ad29dece441-konnectivity-ca\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-modprobe-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396573 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-system-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-hostroot\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396624 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-systemd\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-registration-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-cnibin\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396725 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-cni-binary-copy\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-bin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.396790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-multus\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.397061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.397598 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.397475 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 21:57:14.419581 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.419559 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7dx9h" Nov 02 21:57:14.426314 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.426294 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7dx9h" Nov 02 21:57:14.475899 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.475827 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:57:14.476239 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476217 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:57:14.476399 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476384 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:57:14.476490 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476425 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:57:14.476641 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476554 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:57:14.476641 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476460 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:57:14.476641 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476501 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sd22g\"" Nov 02 21:57:14.476641 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476426 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:57:14.476641 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476615 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:57:14.476956 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476658 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:57:14.476956 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476832 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:57:14.476956 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476833 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5tk4z\"" Nov 02 21:57:14.476956 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476894 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:57:14.476956 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476924 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:57:14.476956 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476933 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:57:14.477216 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.476975 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:57:14.477216 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477002 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-447sr\"" Nov 02 21:57:14.477216 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477024 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:57:14.477216 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477062 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:57:14.477216 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477106 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4qcf6\"" Nov 02 21:57:14.477216 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477149 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n7kkn\"" Nov 02 21:57:14.477216 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477155 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:57:14.477216 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477189 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:57:14.477595 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477254 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:57:14.477595 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477269 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:57:14.477595 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477295 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:57:14.477595 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477392 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wtxk8\"" Nov 02 21:57:14.477595 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477483 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gf8xp\"" Nov 02 21:57:14.477595 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477487 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:57:14.477895 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477654 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:57:14.477895 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477716 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7q497\"" Nov 02 21:57:14.477895 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.477770 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:57:14.487058 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.487036 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:57:14.497082 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.497207 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-socket-dir-parent\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.497207 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-netns\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.497207 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/735421c1-678c-4cc4-a07e-57ffb0e9ca29-tmp-dir\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.497207 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/527b2ac7-739e-4f53-b553-474cad1eb712-host\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.497207 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10158047-423e-4241-b46d-6ad29dece441-agent-certs\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:14.497207 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-socket-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.497509 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497234 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-systemd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.497509 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-dbus\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:14.497509 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:14.497509 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-var-lib-kubelet\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.497509 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.497509 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nccm2\" (UniqueName: \"kubernetes.io/projected/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-kube-api-access-nccm2\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:14.497509 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.497509 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-env-overrides\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.497509 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdfwb\" (UniqueName: \"kubernetes.io/projected/89a526ea-68b3-40e3-a474-02fa8cec0ab0-kube-api-access-bdfwb\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-netns\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-script-lib\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/527b2ac7-739e-4f53-b553-474cad1eb712-serviceca\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497589 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-kubernetes\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh7g\" (UniqueName: \"kubernetes.io/projected/92283fa1-fe9a-4287-a313-176acb992dc9-kube-api-access-dbh7g\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/527b2ac7-739e-4f53-b553-474cad1eb712-kube-api-access-6rqjx\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497785 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-var-lib-kubelet\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497791 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.497909 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.497898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsdx\" (UniqueName: \"kubernetes.io/projected/735421c1-678c-4cc4-a07e-57ffb0e9ca29-kube-api-access-wtsdx\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-k8s-cni-cncf-io\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498071 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-kubernetes\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a18dfddd-46cd-48c6-898c-0b313c829499-host-slash\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-systemd-units\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.498164 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-node-log\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-conf\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.498240 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret podName:3ee5c83e-ba60-41ef-a638-a77cdff3c824 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:14.998217708 +0000 UTC m=+2.986750421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret") pod "global-pull-secret-syncer-qkl72" (UID: "3ee5c83e-ba60-41ef-a638-a77cdff3c824") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-etc-tuned\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498366 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-conf\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.498370 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498367 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/527b2ac7-739e-4f53-b553-474cad1eb712-serviceca\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-multus-daemon-config\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-etc-kubernetes\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqfz5\" (UniqueName: \"kubernetes.io/projected/79214604-184d-4eb5-b63e-d2a30209835c-kube-api-access-mqfz5\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtt6c\" (UniqueName: \"kubernetes.io/projected/a18dfddd-46cd-48c6-898c-0b313c829499-kube-api-access-dtt6c\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-sys\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-netns\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498639 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j57h\" (UniqueName: \"kubernetes.io/projected/53101ce6-458d-4b8f-ac21-36679d929ba0-kube-api-access-9j57h\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-run\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498750 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/527b2ac7-739e-4f53-b553-474cad1eb712-host\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-sys-fs\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-socket-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.498892 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-socket-dir-parent\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498830 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.498947 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-dbus\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499027 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499055 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-run\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-sys-fs\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-kubelet\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-device-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499188 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/735421c1-678c-4cc4-a07e-57ffb0e9ca29-tmp-dir\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfc7z\" (UniqueName: \"kubernetes.io/projected/67af9038-04a3-4574-96fd-23a3bc52d944-kube-api-access-vfc7z\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-cnibin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-kubelet-config\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysconfig\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-tmp\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-kubelet\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.499462 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499443 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-etc-kubernetes\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500080 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovn-node-metrics-cert\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.500080 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-k8s-cni-cncf-io\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500080 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-lib-modules\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.500080 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499614 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.500080 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499708 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-kubelet\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500080 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499868 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-kubelet-config\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:14.500080 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.500080 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.499969 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysconfig\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.500080 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-device-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500124 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-multus-daemon-config\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-cnibin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-sys\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-multus-certs\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-var-lib-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500309 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-etc-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-log-socket\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-netd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-config\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.500455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500442 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-host\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10158047-423e-4241-b46d-6ad29dece441-konnectivity-ca\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-modprobe-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-system-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-hostroot\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500627 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-slash\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-hostroot\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-systemd\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500769 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-lib-modules\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-registration-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500819 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-systemd\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-cnibin\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-cni-binary-copy\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500860 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-host\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.500916 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-bin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.500974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-multus\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-ovn\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-conf-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501067 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-bin\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/735421c1-678c-4cc4-a07e-57ffb0e9ca29-hosts-file\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501119 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-multus-certs\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a18dfddd-46cd-48c6-898c-0b313c829499-iptables-alerter-script\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501270 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501297 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-system-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-bin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501360 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10158047-423e-4241-b46d-6ad29dece441-konnectivity-ca\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-registration-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.501698 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501420 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-multus\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.502536 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-conf-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.502536 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501431 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/735421c1-678c-4cc4-a07e-57ffb0e9ca29-hosts-file\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.502536 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501445 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-cnibin\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.502536 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501461 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-modprobe-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.502536 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.501889 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.502536 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.502336 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-cni-binary-copy\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.503150 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.503128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-etc-tuned\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.503398 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.503379 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10158047-423e-4241-b46d-6ad29dece441-agent-certs\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:14.503691 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.503641 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-tmp\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.507914 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.507887 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbh7g\" (UniqueName: \"kubernetes.io/projected/92283fa1-fe9a-4287-a313-176acb992dc9-kube-api-access-dbh7g\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.508132 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.508111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/527b2ac7-739e-4f53-b553-474cad1eb712-kube-api-access-6rqjx\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.508611 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.508564 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfc7z\" (UniqueName: \"kubernetes.io/projected/67af9038-04a3-4574-96fd-23a3bc52d944-kube-api-access-vfc7z\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.508761 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.508743 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j57h\" (UniqueName: \"kubernetes.io/projected/53101ce6-458d-4b8f-ac21-36679d929ba0-kube-api-access-9j57h\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.509343 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.509311 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsdx\" (UniqueName: \"kubernetes.io/projected/735421c1-678c-4cc4-a07e-57ffb0e9ca29-kube-api-access-wtsdx\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.509985 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.509965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqfz5\" (UniqueName: \"kubernetes.io/projected/79214604-184d-4eb5-b63e-d2a30209835c-kube-api-access-mqfz5\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.538915 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.538833 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"22d5d23fd92916dc8b8ecfbd0ec29fad","Type":"ContainerStarted","Data":"4f82fb253d3e5ce43f20e9f3437d710ed427d55dde183183327a8196762bbdfe"} Nov 02 21:57:14.540235 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.540212 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"ad3b40d52cd903e26b09603b8260563a","Type":"ContainerStarted","Data":"f172b60e0b85874cb3ddda149bd16d5149c21eaef57032b0da728586f2b97831"} Nov 02 21:57:14.602436 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a18dfddd-46cd-48c6-898c-0b313c829499-iptables-alerter-script\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.602581 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-systemd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.602581 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nccm2\" (UniqueName: \"kubernetes.io/projected/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-kube-api-access-nccm2\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:14.602581 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.602581 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-env-overrides\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.602581 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfwb\" (UniqueName: \"kubernetes.io/projected/89a526ea-68b3-40e3-a474-02fa8cec0ab0-kube-api-access-bdfwb\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.602822 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-netns\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.602822 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602616 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-script-lib\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.602822 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:14.602822 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a18dfddd-46cd-48c6-898c-0b313c829499-host-slash\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.602822 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-systemd-units\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.602822 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602731 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-node-log\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.602822 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtt6c\" (UniqueName: \"kubernetes.io/projected/a18dfddd-46cd-48c6-898c-0b313c829499-kube-api-access-dtt6c\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.602822 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:14.602822 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-kubelet\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovn-node-metrics-cert\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-var-lib-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-etc-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-log-socket\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-netd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.602984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-config\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.603003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-slash\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.603017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.603034 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-ovn\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.603049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-bin\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.603120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-bin\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.603588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a18dfddd-46cd-48c6-898c-0b313c829499-iptables-alerter-script\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.603693 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.603665 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-systemd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.603991 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.603972 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.604365 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-env-overrides\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.604608 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-netns\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.605030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-script-lib\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.605098 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.605139 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs podName:5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a nodeName:}" failed. No retries permitted until 2026-11-02 21:57:15.10512684 +0000 UTC m=+3.093659532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs") pod "network-metrics-daemon-4l48q" (UID: "5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.605300 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a18dfddd-46cd-48c6-898c-0b313c829499-host-slash\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.605342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-systemd-units\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.605379 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-node-log\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.605690 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.605738 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-kubelet\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.606096 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.606129 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-slash\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.606149 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-var-lib-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.606147 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-etc-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.606195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-ovn\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.606204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-netd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606231 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.606211 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-log-socket\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.606929 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.606370 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-config\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.609193 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.609170 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovn-node-metrics-cert\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.611439 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.611414 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:57:14.611535 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.611445 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:57:14.611535 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.611460 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvjzw for pod openshift-network-diagnostics/network-check-target-pft59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:14.611535 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:14.611515 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw podName:c5023155-869d-44af-9168-7a44f3cdc1a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:15.111499256 +0000 UTC m=+3.100031963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pvjzw" (UniqueName: "kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw") pod "network-check-target-pft59" (UID: "c5023155-869d-44af-9168-7a44f3cdc1a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:14.613530 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.613503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfwb\" (UniqueName: \"kubernetes.io/projected/89a526ea-68b3-40e3-a474-02fa8cec0ab0-kube-api-access-bdfwb\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.613621 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.613536 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nccm2\" (UniqueName: \"kubernetes.io/projected/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-kube-api-access-nccm2\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:14.614999 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.614979 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtt6c\" (UniqueName: \"kubernetes.io/projected/a18dfddd-46cd-48c6-898c-0b313c829499-kube-api-access-dtt6c\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.685101 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.685063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:14.693378 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.693354 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dg6dc" Nov 02 21:57:14.696634 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:14.696609 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527b2ac7_739e_4f53_b553_474cad1eb712.slice/crio-057d7893582ccea742641ed31efdd53b0566dc7cf383a96c59161bea63f80c44 WatchSource:0}: Error finding container 057d7893582ccea742641ed31efdd53b0566dc7cf383a96c59161bea63f80c44: Status 404 returned error can't find the container with id 057d7893582ccea742641ed31efdd53b0566dc7cf383a96c59161bea63f80c44 Nov 02 21:57:14.703346 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:14.703321 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79214604_184d_4eb5_b63e_d2a30209835c.slice/crio-f4e317226fd6943a30f9538578bab99ade5114f94aa39e42bb043d3b8df65908 WatchSource:0}: Error finding container f4e317226fd6943a30f9538578bab99ade5114f94aa39e42bb043d3b8df65908: Status 404 returned error can't find the container with id f4e317226fd6943a30f9538578bab99ade5114f94aa39e42bb043d3b8df65908 Nov 02 21:57:14.704447 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.704428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:14.710569 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.710542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:14.715176 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:14.715152 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10158047_423e_4241_b46d_6ad29dece441.slice/crio-0fcecc27505fe39fd7d70ff9265661b66b4a61bcdf534e13e2fc9200b8450041 WatchSource:0}: Error finding container 0fcecc27505fe39fd7d70ff9265661b66b4a61bcdf534e13e2fc9200b8450041: Status 404 returned error can't find the container with id 0fcecc27505fe39fd7d70ff9265661b66b4a61bcdf534e13e2fc9200b8450041 Nov 02 21:57:14.717855 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.717837 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:14.721275 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:14.721253 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92283fa1_fe9a_4287_a313_176acb992dc9.slice/crio-23415206da6bed305f7b44294416251313c4eb77104dcca7ecfdb990d8a43fea WatchSource:0}: Error finding container 23415206da6bed305f7b44294416251313c4eb77104dcca7ecfdb990d8a43fea: Status 404 returned error can't find the container with id 23415206da6bed305f7b44294416251313c4eb77104dcca7ecfdb990d8a43fea Nov 02 21:57:14.729200 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:14.729177 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735421c1_678c_4cc4_a07e_57ffb0e9ca29.slice/crio-9486167f074d3846cd429fd96ca030d5e09c311bdb24e08c58d0d51fab754a3e WatchSource:0}: Error finding container 9486167f074d3846cd429fd96ca030d5e09c311bdb24e08c58d0d51fab754a3e: Status 404 returned error can't find the container with id 9486167f074d3846cd429fd96ca030d5e09c311bdb24e08c58d0d51fab754a3e Nov 02 21:57:14.730547 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.730528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:14.740907 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:14.740886 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67af9038_04a3_4574_96fd_23a3bc52d944.slice/crio-8a4b3104ff10beff33d55a15173ed460e1cffc6728a6f7dfa19bc300adf4bd7d WatchSource:0}: Error finding container 8a4b3104ff10beff33d55a15173ed460e1cffc6728a6f7dfa19bc300adf4bd7d: Status 404 returned error can't find the container with id 8a4b3104ff10beff33d55a15173ed460e1cffc6728a6f7dfa19bc300adf4bd7d Nov 02 21:57:14.746455 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.746423 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:14.753285 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.753266 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:14.756722 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:14.756694 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18dfddd_46cd_48c6_898c_0b313c829499.slice/crio-e48fe2500e9a7aeaf9c73f1b49aaa3991ba80ca36e76be503ef8c40d8fdf1642 WatchSource:0}: Error finding container e48fe2500e9a7aeaf9c73f1b49aaa3991ba80ca36e76be503ef8c40d8fdf1642: Status 404 returned error can't find the container with id e48fe2500e9a7aeaf9c73f1b49aaa3991ba80ca36e76be503ef8c40d8fdf1642 Nov 02 21:57:14.763362 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:14.763341 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53101ce6_458d_4b8f_ac21_36679d929ba0.slice/crio-d026b8e0f166199936add29edb8a6e8e091375f4b0bbcea6c2fdcecb1f415ee5 WatchSource:0}: Error finding container d026b8e0f166199936add29edb8a6e8e091375f4b0bbcea6c2fdcecb1f415ee5: Status 404 returned error can't find the container with id d026b8e0f166199936add29edb8a6e8e091375f4b0bbcea6c2fdcecb1f415ee5 Nov 02 21:57:14.768151 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.768132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:14.778790 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:14.778769 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a526ea_68b3_40e3_a474_02fa8cec0ab0.slice/crio-07389bee425d03feac29195ca6f0778dd11eef65839eddc74e9b6ace5fa2ec0d WatchSource:0}: Error finding container 07389bee425d03feac29195ca6f0778dd11eef65839eddc74e9b6ace5fa2ec0d: Status 404 returned error can't find the container with id 07389bee425d03feac29195ca6f0778dd11eef65839eddc74e9b6ace5fa2ec0d Nov 02 21:57:14.845992 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:14.845966 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:15.006818 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.006793 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:15.006949 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:15.006930 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:15.007003 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:15.006992 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret podName:3ee5c83e-ba60-41ef-a638-a77cdff3c824 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:16.0069761 +0000 UTC m=+3.995508797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret") pod "global-pull-secret-syncer-qkl72" (UID: "3ee5c83e-ba60-41ef-a638-a77cdff3c824") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:15.107824 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.107740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:15.107987 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:15.107860 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:15.107987 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:15.107922 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs podName:5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a nodeName:}" failed. No retries permitted until 2026-11-02 21:57:16.107907535 +0000 UTC m=+4.096440227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs") pod "network-metrics-daemon-4l48q" (UID: "5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:15.209225 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.208624 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:15.209225 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:15.208797 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:57:15.209225 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:15.208815 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:57:15.209225 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:15.208828 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvjzw for pod openshift-network-diagnostics/network-check-target-pft59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:15.209225 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:15.208882 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw podName:c5023155-869d-44af-9168-7a44f3cdc1a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:16.208865056 +0000 UTC m=+4.197397758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvjzw" (UniqueName: "kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw") pod "network-check-target-pft59" (UID: "c5023155-869d-44af-9168-7a44f3cdc1a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:15.291991 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.291766 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:15.429150 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.429039 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:52:14 +0000 UTC" deadline="2028-06-16 12:12:37.232320109 +0000 UTC" Nov 02 21:57:15.429150 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.429097 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14198h15m21.803227572s" Nov 02 21:57:15.527810 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.527325 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:15.527810 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:15.527453 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:15.565035 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.564995 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerStarted","Data":"07389bee425d03feac29195ca6f0778dd11eef65839eddc74e9b6ace5fa2ec0d"} Nov 02 21:57:15.572125 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.572089 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqhnn" event={"ID":"a18dfddd-46cd-48c6-898c-0b313c829499","Type":"ContainerStarted","Data":"e48fe2500e9a7aeaf9c73f1b49aaa3991ba80ca36e76be503ef8c40d8fdf1642"} Nov 02 21:57:15.579723 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.579689 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" event={"ID":"67af9038-04a3-4574-96fd-23a3bc52d944","Type":"ContainerStarted","Data":"8a4b3104ff10beff33d55a15173ed460e1cffc6728a6f7dfa19bc300adf4bd7d"} Nov 02 21:57:15.589495 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.589456 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rszqx" event={"ID":"92283fa1-fe9a-4287-a313-176acb992dc9","Type":"ContainerStarted","Data":"23415206da6bed305f7b44294416251313c4eb77104dcca7ecfdb990d8a43fea"} Nov 02 21:57:15.595786 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.595758 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxdmr" event={"ID":"10158047-423e-4241-b46d-6ad29dece441","Type":"ContainerStarted","Data":"0fcecc27505fe39fd7d70ff9265661b66b4a61bcdf534e13e2fc9200b8450041"} Nov 02 21:57:15.597295 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.597270 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dg6dc" event={"ID":"79214604-184d-4eb5-b63e-d2a30209835c","Type":"ContainerStarted","Data":"f4e317226fd6943a30f9538578bab99ade5114f94aa39e42bb043d3b8df65908"} Nov 02 21:57:15.603432 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.603404 2578 generic.go:356] "Generic (PLEG): container finished" podID="22d5d23fd92916dc8b8ecfbd0ec29fad" containerID="511949b91d25abc3ed0673e142b690d4197ebd732ea404b1afe5f3e1d3b6735f" exitCode=0 Nov 02 21:57:15.603538 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.603507 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"22d5d23fd92916dc8b8ecfbd0ec29fad","Type":"ContainerDied","Data":"511949b91d25abc3ed0673e142b690d4197ebd732ea404b1afe5f3e1d3b6735f"} Nov 02 21:57:15.609355 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.609193 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jc58" event={"ID":"53101ce6-458d-4b8f-ac21-36679d929ba0","Type":"ContainerStarted","Data":"d026b8e0f166199936add29edb8a6e8e091375f4b0bbcea6c2fdcecb1f415ee5"} Nov 02 21:57:15.611927 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.611845 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kclzj" event={"ID":"735421c1-678c-4cc4-a07e-57ffb0e9ca29","Type":"ContainerStarted","Data":"9486167f074d3846cd429fd96ca030d5e09c311bdb24e08c58d0d51fab754a3e"} Nov 02 21:57:15.626186 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:15.626130 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jg7pl" event={"ID":"527b2ac7-739e-4f53-b553-474cad1eb712","Type":"ContainerStarted","Data":"057d7893582ccea742641ed31efdd53b0566dc7cf383a96c59161bea63f80c44"} Nov 02 21:57:16.014144 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.013792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:16.014310 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.014206 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:16.014310 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.014270 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret podName:3ee5c83e-ba60-41ef-a638-a77cdff3c824 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:18.014253625 +0000 UTC m=+6.002786330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret") pod "global-pull-secret-syncer-qkl72" (UID: "3ee5c83e-ba60-41ef-a638-a77cdff3c824") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:16.115062 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.115009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:16.115241 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.115134 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:16.115241 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.115180 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs podName:5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a nodeName:}" failed. No retries permitted until 2026-11-02 21:57:18.115168401 +0000 UTC m=+6.103701106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs") pod "network-metrics-daemon-4l48q" (UID: "5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:16.216496 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.216461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:16.216903 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.216883 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:57:16.216979 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.216911 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:57:16.216979 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.216923 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvjzw for pod openshift-network-diagnostics/network-check-target-pft59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:16.217086 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.216985 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw podName:c5023155-869d-44af-9168-7a44f3cdc1a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:18.216968726 +0000 UTC m=+6.205501431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvjzw" (UniqueName: "kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw") pod "network-check-target-pft59" (UID: "c5023155-869d-44af-9168-7a44f3cdc1a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:16.431515 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.431269 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:52:14 +0000 UTC" deadline="2028-05-20 10:57:33.510895714 +0000 UTC" Nov 02 21:57:16.431515 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.431329 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13549h0m17.079571262s" Nov 02 21:57:16.530397 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.529700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:16.530397 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.529823 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:16.530397 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.530204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:16.530397 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:16.530287 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:16.639623 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.638907 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"ad3b40d52cd903e26b09603b8260563a","Type":"ContainerStarted","Data":"1166cfd8fcd241c6ad2e73f9205950b8eef83d6392b545e87b2103d0f0239b22"} Nov 02 21:57:16.642606 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.642577 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"22d5d23fd92916dc8b8ecfbd0ec29fad","Type":"ContainerStarted","Data":"ca389cc7a2acb725db713497cfd983cae2ee99e96473fba7d5a94432a356ccee"} Nov 02 21:57:16.667206 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:16.667150 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" podStartSLOduration=3.66713191 podStartE2EDuration="3.66713191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:16.653491207 +0000 UTC m=+4.642023924" watchObservedRunningTime="2026-11-02 21:57:16.66713191 +0000 UTC m=+4.655664627" Nov 02 21:57:17.528670 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:17.528080 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:17.528670 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:17.528221 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:18.030874 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:18.030306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:18.030874 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.030455 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:18.030874 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.030517 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret podName:3ee5c83e-ba60-41ef-a638-a77cdff3c824 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:22.030499154 +0000 UTC m=+10.019031869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret") pod "global-pull-secret-syncer-qkl72" (UID: "3ee5c83e-ba60-41ef-a638-a77cdff3c824") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:18.131569 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:18.131476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:18.131763 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.131746 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:18.131852 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.131814 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs podName:5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a nodeName:}" failed. No retries permitted until 2026-11-02 21:57:22.131795943 +0000 UTC m=+10.120328640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs") pod "network-metrics-daemon-4l48q" (UID: "5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:18.232471 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:18.232434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:18.232642 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.232607 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:57:18.232642 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.232625 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:57:18.232786 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.232646 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvjzw for pod openshift-network-diagnostics/network-check-target-pft59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:18.232786 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.232748 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw podName:c5023155-869d-44af-9168-7a44f3cdc1a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:22.232727443 +0000 UTC m=+10.221260147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvjzw" (UniqueName: "kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw") pod "network-check-target-pft59" (UID: "c5023155-869d-44af-9168-7a44f3cdc1a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:18.528355 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:18.528272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:18.528510 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.528406 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:18.528815 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:18.528792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:18.529191 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:18.528905 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:18.664835 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:18.664799 2578 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="4938cd3b82a6b6f280ffa365de8be87aa9d63aab5e374a49817a9373dfc9ee55" exitCode=0 Nov 02 21:57:18.664984 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:18.664865 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jc58" event={"ID":"53101ce6-458d-4b8f-ac21-36679d929ba0","Type":"ContainerDied","Data":"4938cd3b82a6b6f280ffa365de8be87aa9d63aab5e374a49817a9373dfc9ee55"} Nov 02 21:57:18.688729 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:18.688666 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" podStartSLOduration=5.68865357 podStartE2EDuration="5.68865357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:16.668294485 +0000 UTC m=+4.656827200" watchObservedRunningTime="2026-11-02 21:57:18.68865357 +0000 UTC m=+6.677186280" Nov 02 21:57:19.528427 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:19.528086 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:19.528427 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:19.528230 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:19.670826 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:19.670752 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxdmr" event={"ID":"10158047-423e-4241-b46d-6ad29dece441","Type":"ContainerStarted","Data":"0881509e5e565321cd65e210071d1b59add29c29157501f65c527fdfeaaa46a0"} Nov 02 21:57:20.527839 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:20.527779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:20.527949 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:20.527906 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:20.528189 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:20.528169 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:20.528286 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:20.528269 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:20.677776 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:20.677613 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jg7pl" event={"ID":"527b2ac7-739e-4f53-b553-474cad1eb712","Type":"ContainerStarted","Data":"0158be84ab22cdd14b2c02088181892aee3f3b9c89f0e2f83b62e7dd057f1a3c"} Nov 02 21:57:20.681767 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:20.681493 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" event={"ID":"67af9038-04a3-4574-96fd-23a3bc52d944","Type":"ContainerStarted","Data":"e1729b0eb310acd79c24b900a041520c72f03956f8f1903f5e0864e070099da6"} Nov 02 21:57:20.694204 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:20.694147 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zxdmr" podStartSLOduration=4.861422152 podStartE2EDuration="8.694128811s" totalImagesPullingTime="3.832706659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:19.686108008 +0000 UTC m=+7.674640723" watchObservedRunningTime="2026-11-02 21:57:20.694128811 +0000 UTC m=+8.682661528" Nov 02 21:57:20.694585 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:20.694389 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jg7pl" podStartSLOduration=3.582955664 podStartE2EDuration="8.694378652s" totalImagesPullingTime="5.111422988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:20.693909928 +0000 UTC m=+8.682442645" watchObservedRunningTime="2026-11-02 21:57:20.694378652 +0000 UTC m=+8.682911370" Nov 02 21:57:21.528201 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:21.528123 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:21.528351 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:21.528305 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:22.065617 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.065575 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:22.066072 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.065751 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:22.066072 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.065812 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret podName:3ee5c83e-ba60-41ef-a638-a77cdff3c824 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:30.065794719 +0000 UTC m=+18.054327426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret") pod "global-pull-secret-syncer-qkl72" (UID: "3ee5c83e-ba60-41ef-a638-a77cdff3c824") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:22.166630 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.166599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:22.166781 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.166770 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:22.166847 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.166832 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs podName:5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a nodeName:}" failed. No retries permitted until 2026-11-02 21:57:30.16681304 +0000 UTC m=+18.155345753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs") pod "network-metrics-daemon-4l48q" (UID: "5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:22.210662 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.210634 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:57:22.267609 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.267118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:22.267609 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.267252 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:57:22.267609 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.267266 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:57:22.267609 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.267274 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvjzw for pod openshift-network-diagnostics/network-check-target-pft59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:22.267609 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.267317 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw podName:c5023155-869d-44af-9168-7a44f3cdc1a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:30.267305584 +0000 UTC m=+18.255838289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvjzw" (UniqueName: "kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw") pod "network-check-target-pft59" (UID: "c5023155-869d-44af-9168-7a44f3cdc1a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:22.456765 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.453747 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:57:22.210651643Z","UUID":"a97ccb2c-bcde-43e4-b231-3803a86c2bdd","Handler":null,"Name":"","Endpoint":""} Nov 02 21:57:22.456765 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.455627 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:57:22.456765 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.455657 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:57:22.530268 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.529806 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:22.530268 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.529937 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:22.530268 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.530042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:22.530268 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:22.530150 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:22.692593 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.692551 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" event={"ID":"67af9038-04a3-4574-96fd-23a3bc52d944","Type":"ContainerStarted","Data":"e1ebf99199210ebda965193dd13484bde835038f05e0e5998115f025d275ac90"} Nov 02 21:57:22.694510 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.694481 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dg6dc" event={"ID":"79214604-184d-4eb5-b63e-d2a30209835c","Type":"ContainerStarted","Data":"0cfc9a341c65a53ca8e60a72e4f62976be9e0786e1bfbbe0e95a98e2dd7311f5"} Nov 02 21:57:22.696187 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.696162 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kclzj" event={"ID":"735421c1-678c-4cc4-a07e-57ffb0e9ca29","Type":"ContainerStarted","Data":"c7cf15e6089e15b05cc8f34cef46da3fb36ac067257f9d1ad25c66f4657b1511"} Nov 02 21:57:22.712270 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:22.712221 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-dg6dc" podStartSLOduration=3.101815587 podStartE2EDuration="10.712205836s" totalImagesPullingTime="7.610390249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:22.712115358 +0000 UTC m=+10.700648074" watchObservedRunningTime="2026-11-02 21:57:22.712205836 +0000 UTC m=+10.700738548" Nov 02 21:57:23.528321 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:23.528125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:23.528759 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:23.528441 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:23.696398 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:23.696314 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:23.697310 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:23.697103 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:23.700610 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:23.700581 2578 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="debdfc9ca4c3a790b88f6ee1965a5885a1197a020b4080fa917e5754f1c0e459" exitCode=0 Nov 02 21:57:23.700752 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:23.700669 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jc58" event={"ID":"53101ce6-458d-4b8f-ac21-36679d929ba0","Type":"ContainerDied","Data":"debdfc9ca4c3a790b88f6ee1965a5885a1197a020b4080fa917e5754f1c0e459"} Nov 02 21:57:23.704448 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:23.704378 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" event={"ID":"67af9038-04a3-4574-96fd-23a3bc52d944","Type":"ContainerStarted","Data":"e5862fe58d8d3d7aad37a4ded4f81a3b7ba03a811aeac6c5e56c760f2bed5806"} Nov 02 21:57:23.712527 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:23.712486 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kclzj" podStartSLOduration=4.052925624 podStartE2EDuration="11.712470248s" totalImagesPullingTime="7.659544624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:22.725471667 +0000 UTC m=+10.714004360" watchObservedRunningTime="2026-11-02 21:57:23.712470248 +0000 UTC m=+11.701002964" Nov 02 21:57:23.727894 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:23.727835 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" podStartSLOduration=3.473069635 podStartE2EDuration="11.727818128s" totalImagesPullingTime="8.254748493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:23.727338228 +0000 UTC m=+11.715870941" watchObservedRunningTime="2026-11-02 21:57:23.727818128 +0000 UTC m=+11.716350841" Nov 02 21:57:24.530010 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:24.529975 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:24.530403 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:24.530102 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:24.530466 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:24.530452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:24.530554 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:24.530534 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:24.708741 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:24.708707 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rszqx" event={"ID":"92283fa1-fe9a-4287-a313-176acb992dc9","Type":"ContainerStarted","Data":"3ba0ba133058b64d6e2dbb65ef2c29eed63272a785ae200c53290280dff26d87"} Nov 02 21:57:24.711801 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:24.711777 2578 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="7062bc8a5e7d839f7df33c0d48945671721ade022ea0735011e68ed835a72f2b" exitCode=0 Nov 02 21:57:24.711922 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:24.711839 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jc58" event={"ID":"53101ce6-458d-4b8f-ac21-36679d929ba0","Type":"ContainerDied","Data":"7062bc8a5e7d839f7df33c0d48945671721ade022ea0735011e68ed835a72f2b"} Nov 02 21:57:24.714221 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:24.714158 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqhnn" event={"ID":"a18dfddd-46cd-48c6-898c-0b313c829499","Type":"ContainerStarted","Data":"85f951bba4b0845ba0ee2c4a813efce63a217338b9ed7056113f74d933d55700"} Nov 02 21:57:24.725012 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:24.724967 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rszqx" podStartSLOduration=2.864702496 podStartE2EDuration="12.72495489s" totalImagesPullingTime="9.860252394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:24.724713558 +0000 UTC m=+12.713246269" watchObservedRunningTime="2026-11-02 21:57:24.72495489 +0000 UTC m=+12.713487604" Nov 02 21:57:24.755993 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:24.755937 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lqhnn" podStartSLOduration=4.699338161 podStartE2EDuration="12.75592162s" totalImagesPullingTime="8.056583459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:24.755580882 +0000 UTC m=+12.744113596" watchObservedRunningTime="2026-11-02 21:57:24.75592162 +0000 UTC m=+12.744454334" Nov 02 21:57:25.527388 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:25.527356 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:25.527559 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:25.527480 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:26.528336 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:26.528302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:26.529104 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:26.528418 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:26.529164 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:26.529108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:26.529217 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:26.529203 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:26.732333 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:26.732291 2578 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="1a885ed94fb6238b4b69ab3750ae929fb46d9d6c63ae3cc219f47f859764b78a" exitCode=0 Nov 02 21:57:26.732333 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:26.732337 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jc58" event={"ID":"53101ce6-458d-4b8f-ac21-36679d929ba0","Type":"ContainerDied","Data":"1a885ed94fb6238b4b69ab3750ae929fb46d9d6c63ae3cc219f47f859764b78a"} Nov 02 21:57:27.527640 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:27.527606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:27.527813 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:27.527761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:28.531437 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:28.531250 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:28.532054 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:28.531304 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:28.532054 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:28.531510 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:28.532054 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:28.531574 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:28.739285 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:28.739197 2578 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="a04b6307e098ad81ad63742a71d0061fd615648316f833dd412b037d2d63697b" exitCode=0 Nov 02 21:57:28.739285 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:28.739232 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jc58" event={"ID":"53101ce6-458d-4b8f-ac21-36679d929ba0","Type":"ContainerDied","Data":"a04b6307e098ad81ad63742a71d0061fd615648316f833dd412b037d2d63697b"} Nov 02 21:57:29.528234 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.528034 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:29.528436 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:29.528269 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:29.743644 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.743619 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 21:57:29.744088 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.743941 2578 generic.go:356] "Generic (PLEG): container finished" podID="89a526ea-68b3-40e3-a474-02fa8cec0ab0" containerID="0c2690e17dc40dc49f1100d2bfb3f95405598913759a45096722f0e0505532ab" exitCode=1 Nov 02 21:57:29.744088 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.744015 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerStarted","Data":"0a3a53940e5a9a10a59bd2f3fa02ee1b3fea6e28ea1b28e6c51d310e94cbf5fa"} Nov 02 21:57:29.744209 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.744099 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerStarted","Data":"e66a6cbcf5b95710b1dd678dd3d4bf495ae61b89acd2efca87efcae6d6afc530"} Nov 02 21:57:29.744209 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.744152 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerStarted","Data":"128215681a3ef1ee7a47b843cf559aee532cdb5a7d1fad626a2ebcde3162cbee"} Nov 02 21:57:29.744209 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.744203 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerStarted","Data":"9d15dd11affca85a4429f3a49ccf8c8c4ce89878694d786a0dbd1e73b910ec6c"} Nov 02 21:57:29.744364 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.744251 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerDied","Data":"0c2690e17dc40dc49f1100d2bfb3f95405598913759a45096722f0e0505532ab"} Nov 02 21:57:29.744364 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.744302 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerStarted","Data":"0392e17cc140c4de6f88237910b438064530639f4d2e31dbe94b13b3bad5ba16"} Nov 02 21:57:29.746826 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.746806 2578 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="b2f2e5bc78b55ed667e333c9d6bed91cecb10569f36aeb418617002544acdfc0" exitCode=0 Nov 02 21:57:29.746928 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:29.746837 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jc58" event={"ID":"53101ce6-458d-4b8f-ac21-36679d929ba0","Type":"ContainerDied","Data":"b2f2e5bc78b55ed667e333c9d6bed91cecb10569f36aeb418617002544acdfc0"} Nov 02 21:57:30.131073 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:30.131043 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:30.131238 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.131146 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:30.131238 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.131191 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret podName:3ee5c83e-ba60-41ef-a638-a77cdff3c824 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:46.131178873 +0000 UTC m=+34.119711578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret") pod "global-pull-secret-syncer-qkl72" (UID: "3ee5c83e-ba60-41ef-a638-a77cdff3c824") : object "kube-system"/"original-pull-secret" not registered Nov 02 21:57:30.232082 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:30.232049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:30.232229 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.232182 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:30.232273 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.232239 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs podName:5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a nodeName:}" failed. No retries permitted until 2026-11-02 21:57:46.232226177 +0000 UTC m=+34.220758873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs") pod "network-metrics-daemon-4l48q" (UID: "5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 21:57:30.333022 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:30.332992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:30.333176 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.333112 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 21:57:30.333176 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.333125 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 21:57:30.333176 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.333134 2578 projected.go:196] Error preparing data for projected volume kube-api-access-pvjzw for pod openshift-network-diagnostics/network-check-target-pft59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:30.333176 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.333174 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw podName:c5023155-869d-44af-9168-7a44f3cdc1a3 nodeName:}" failed. No retries permitted until 2026-11-02 21:57:46.333163242 +0000 UTC m=+34.321695935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pvjzw" (UniqueName: "kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw") pod "network-check-target-pft59" (UID: "c5023155-869d-44af-9168-7a44f3cdc1a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 21:57:30.528148 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:30.528076 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:30.528283 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.528204 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:30.528283 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:30.528237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:30.528358 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:30.528286 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:30.752130 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:30.752094 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6jc58" event={"ID":"53101ce6-458d-4b8f-ac21-36679d929ba0","Type":"ContainerStarted","Data":"9d8b424d4668e3c60bac2edbca1ea8e639f062cc93be0ac0cec92d145504eb66"} Nov 02 21:57:30.771871 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:30.771829 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6jc58" podStartSLOduration=7.951070628 podStartE2EDuration="18.771816967s" totalImagesPullingTime="10.820746339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:30.771041642 +0000 UTC m=+18.759574357" watchObservedRunningTime="2026-11-02 21:57:30.771816967 +0000 UTC m=+18.760349681" Nov 02 21:57:31.528280 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:31.528250 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:31.528407 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:31.528385 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:31.756398 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:31.756323 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 21:57:31.756772 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:31.756627 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerStarted","Data":"c4fb2989ae4b15e208a4dcdfd4002f9da2068e2672ce53091dc125091bca2e51"} Nov 02 21:57:32.528864 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:32.528832 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:32.529045 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:32.528911 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:32.529045 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:32.529010 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:32.529154 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:32.529110 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:33.527924 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:33.527891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:33.528337 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:33.528001 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:33.764595 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:33.764438 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 21:57:33.764961 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:33.764938 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerStarted","Data":"b1f245a81ba5631bcde614322d12c4073dcfc4e24add338456bb201840a0158b"} Nov 02 21:57:33.765255 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:33.765234 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:33.765361 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:33.765352 2578 scope.go:122] "RemoveContainer" containerID="0c2690e17dc40dc49f1100d2bfb3f95405598913759a45096722f0e0505532ab" Nov 02 21:57:33.782144 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:33.782119 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:34.527787 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:34.527752 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:34.527975 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:34.527761 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:34.527975 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:34.527881 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:34.527975 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:34.527930 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:34.771189 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:34.771161 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 21:57:34.771991 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:34.771961 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" event={"ID":"89a526ea-68b3-40e3-a474-02fa8cec0ab0","Type":"ContainerStarted","Data":"e58ce069656e062c781e13bf83f6537cee881dc87bc92954c95cb1e9ee011e8e"} Nov 02 21:57:34.772594 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:34.772550 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:34.772717 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:34.772635 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:34.794535 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:34.792167 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:34.800570 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:34.800515 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" podStartSLOduration=8.585984792 podStartE2EDuration="22.800497118s" totalImagesPullingTime="14.214512326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:34.80017277 +0000 UTC m=+22.788705488" watchObservedRunningTime="2026-11-02 21:57:34.800497118 +0000 UTC m=+22.789029833" Nov 02 21:57:35.306653 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:35.306579 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4l48q"] Nov 02 21:57:35.306862 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:35.306760 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:35.306921 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:35.306861 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:35.307241 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:35.307219 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qkl72"] Nov 02 21:57:35.307320 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:35.307308 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:35.307420 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:35.307402 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:35.308960 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:35.308938 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pft59"] Nov 02 21:57:35.309063 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:35.309018 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:35.309112 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:35.309087 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:35.726757 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:35.726719 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:35.727266 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:35.726857 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:57:35.727372 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:35.727350 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:37.527951 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:37.527925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:37.528364 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:37.527925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:37.528364 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:37.528016 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:37.528364 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:37.527925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:37.528364 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:37.528087 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:37.528364 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:37.528172 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:39.528333 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:39.528302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:39.528333 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:39.528340 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:39.528764 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:39.528302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:39.528764 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:39.528408 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pft59" podUID="c5023155-869d-44af-9168-7a44f3cdc1a3" Nov 02 21:57:39.528764 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:39.528507 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qkl72" podUID="3ee5c83e-ba60-41ef-a638-a77cdff3c824" Nov 02 21:57:39.528764 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:39.528552 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4l48q" podUID="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" Nov 02 21:57:40.872304 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:40.872275 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeReady" Nov 02 21:57:40.872671 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:40.872390 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 21:57:41.527647 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.527429 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:41.527807 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.527479 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:41.527865 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.527482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:41.531967 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.531944 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:57:41.532084 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.532008 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:57:41.533256 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.533237 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:57:41.533256 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.533244 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6t4jd\"" Nov 02 21:57:41.533394 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.533300 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vjf48\"" Nov 02 21:57:41.533394 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.533314 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9dzjg\"" Nov 02 21:57:41.533461 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:41.533392 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:57:45.690562 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:45.690533 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kclzj_735421c1-678c-4cc4-a07e-57ffb0e9ca29/dns-node-resolver/0.log" Nov 02 21:57:46.143943 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.143904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:46.147576 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.147558 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:46.244815 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.244781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:46.244953 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:46.244904 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 21:57:46.244991 ip-10-0-136-9 kubenswrapper[2578]: E1102 21:57:46.244954 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs podName:5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a nodeName:}" failed. No retries permitted until 2026-11-02 21:58:18.244942121 +0000 UTC m=+66.233474818 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs") pod "network-metrics-daemon-4l48q" (UID: "5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a") : secret "metrics-daemon-secret" not found Nov 02 21:57:46.344754 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.344717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:46.345910 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.345888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:46.348198 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.348181 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:46.640263 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.640230 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:46.775278 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.775207 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qkl72"] Nov 02 21:57:46.778623 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:46.778598 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee5c83e_ba60_41ef_a638_a77cdff3c824.slice/crio-15532698420d72d2aafd4a0b6f08d3bab05526ece36c670e8dfc3bad6565a8a5 WatchSource:0}: Error finding container 15532698420d72d2aafd4a0b6f08d3bab05526ece36c670e8dfc3bad6565a8a5: Status 404 returned error can't find the container with id 15532698420d72d2aafd4a0b6f08d3bab05526ece36c670e8dfc3bad6565a8a5 Nov 02 21:57:46.799052 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.799020 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qkl72" event={"ID":"3ee5c83e-ba60-41ef-a638-a77cdff3c824","Type":"ContainerStarted","Data":"15532698420d72d2aafd4a0b6f08d3bab05526ece36c670e8dfc3bad6565a8a5"} Nov 02 21:57:46.862792 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.862770 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jg7pl_527b2ac7-739e-4f53-b553-474cad1eb712/node-ca/0.log" Nov 02 21:57:46.910799 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:46.910771 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pft59"] Nov 02 21:57:46.913719 ip-10-0-136-9 kubenswrapper[2578]: W1102 21:57:46.913692 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5023155_869d_44af_9168_7a44f3cdc1a3.slice/crio-f32627a3012229ad56a0b727ad7cd27cd93702777ef1e483665ca47e38333f22 WatchSource:0}: Error finding container f32627a3012229ad56a0b727ad7cd27cd93702777ef1e483665ca47e38333f22: Status 404 returned error can't find the container with id f32627a3012229ad56a0b727ad7cd27cd93702777ef1e483665ca47e38333f22 Nov 02 21:57:47.802835 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:47.802765 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pft59" event={"ID":"c5023155-869d-44af-9168-7a44f3cdc1a3","Type":"ContainerStarted","Data":"f32627a3012229ad56a0b727ad7cd27cd93702777ef1e483665ca47e38333f22"} Nov 02 21:57:50.811879 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:50.811843 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pft59" event={"ID":"c5023155-869d-44af-9168-7a44f3cdc1a3","Type":"ContainerStarted","Data":"84ed3d7ce4e473807473ddaa6dc61ad7d196bc40d028e27152c5ab8f532686da"} Nov 02 21:57:50.812350 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:50.811958 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:50.826917 ip-10-0-136-9 kubenswrapper[2578]: I1102 21:57:50.826867 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pft59" podStartSLOduration=35.18770165 podStartE2EDuration="38.826851099s" totalImagesPullingTime="3.639149449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:57:50.826584272 +0000 UTC m=+38.815116985" watchObservedRunningTime="2026-11-02 21:57:50.826851099 +0000 UTC m=+38.815383814" Nov 02 21:57:52.012425 ip-10-0-136-9 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 21:57:52.022179 ip-10-0-136-9 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 21:57:52.022347 ip-10-0-136-9 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 21:57:52.022397 ip-10-0-136-9 systemd[1]: kubelet.service: Consumed 2.038s CPU time. Nov 02 21:57:52.033318 ip-10-0-136-9 systemd[1]: Starting Kubernetes Kubelet... Nov 02 21:57:52.103921 ip-10-0-136-9 kubenswrapper[4309]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:57:52.103921 ip-10-0-136-9 kubenswrapper[4309]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 21:57:52.103921 ip-10-0-136-9 kubenswrapper[4309]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:57:52.103921 ip-10-0-136-9 kubenswrapper[4309]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 21:57:52.106282 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106194 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:52.106282 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106278 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:52.106282 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106283 4309 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:52.106282 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106288 4309 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106293 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106298 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106304 4309 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106310 4309 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106314 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106318 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106322 4309 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106326 4309 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106330 4309 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106334 4309 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106338 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106342 4309 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106346 4309 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106350 4309 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106354 4309 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106363 4309 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106367 4309 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106371 4309 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:52.106557 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106375 4309 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106379 4309 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106383 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106387 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106391 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106395 4309 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106399 4309 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106404 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106407 4309 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106412 4309 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106416 4309 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106420 4309 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106425 4309 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106429 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106434 4309 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106439 4309 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106443 4309 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106447 4309 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106452 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106456 4309 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:52.107466 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106461 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106465 4309 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106469 4309 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106473 4309 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106477 4309 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106482 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106486 4309 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106492 4309 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106496 4309 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106500 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106508 4309 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106512 4309 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106516 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106521 4309 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106525 4309 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106529 4309 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106533 4309 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106538 4309 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106542 4309 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106547 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:52.108364 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106552 4309 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106556 4309 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106560 4309 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106564 4309 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106568 4309 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106573 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106578 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106582 4309 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106586 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106590 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106595 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106599 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106604 4309 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106608 4309 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106613 4309 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106617 4309 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106621 4309 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106625 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106630 4309 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:52.109262 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106635 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106641 4309 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106647 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106652 4309 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106658 4309 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106663 4309 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106668 4309 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106689 4309 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106695 4309 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106699 4309 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106703 4309 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106707 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106712 4309 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106717 4309 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106721 4309 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106725 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106730 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106734 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106738 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:52.110159 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106743 4309 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106748 4309 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106753 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106758 4309 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106762 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.106767 4309 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108057 4309 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108069 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108073 4309 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108079 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108083 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108087 4309 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108092 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108096 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108100 4309 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108105 4309 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108110 4309 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108115 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108120 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:52.111043 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108127 4309 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108134 4309 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108141 4309 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108146 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108151 4309 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108156 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108160 4309 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108165 4309 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108169 4309 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108174 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108178 4309 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108182 4309 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108187 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108191 4309 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108195 4309 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108199 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108204 4309 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108208 4309 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108213 4309 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:52.111904 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108217 4309 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108221 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108226 4309 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108230 4309 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108234 4309 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108238 4309 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108243 4309 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108247 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108251 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108256 4309 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108260 4309 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108264 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108269 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108278 4309 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108282 4309 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108287 4309 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108292 4309 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108296 4309 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108301 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:52.112746 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108305 4309 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108309 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108314 4309 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108318 4309 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108323 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108327 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108331 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108335 4309 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108340 4309 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108344 4309 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108348 4309 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108353 4309 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108357 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108360 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108365 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108369 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108372 4309 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108376 4309 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108380 4309 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108384 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:52.113329 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108388 4309 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108393 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108397 4309 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108403 4309 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108408 4309 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108412 4309 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108416 4309 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108431 4309 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108436 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108440 4309 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108445 4309 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108449 4309 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108455 4309 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108459 4309 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108464 4309 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108468 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108473 4309 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108477 4309 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108481 4309 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108485 4309 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:52.113859 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108490 4309 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108494 4309 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108498 4309 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108503 4309 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108508 4309 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108512 4309 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108517 4309 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108521 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108525 4309 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108529 4309 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108534 4309 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108538 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108542 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108550 4309 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.108557 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109444 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109462 4309 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109467 4309 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109472 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109485 4309 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:52.114352 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109490 4309 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109494 4309 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109498 4309 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109503 4309 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109508 4309 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109512 4309 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109516 4309 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109521 4309 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109525 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109530 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109534 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109538 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109542 4309 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109547 4309 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109551 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109555 4309 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109559 4309 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109563 4309 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109568 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109572 4309 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:52.114915 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109585 4309 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109589 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109594 4309 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109598 4309 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109602 4309 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109606 4309 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109611 4309 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109615 4309 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109620 4309 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109624 4309 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109628 4309 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109633 4309 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109645 4309 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109649 4309 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109654 4309 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109658 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109662 4309 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109666 4309 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109687 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:52.115457 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109692 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109696 4309 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109701 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109705 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109709 4309 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109714 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109718 4309 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109723 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109728 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109731 4309 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109736 4309 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109742 4309 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109749 4309 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109757 4309 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109761 4309 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109766 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109770 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109774 4309 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109778 4309 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:52.116311 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109783 4309 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109787 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109792 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109796 4309 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109800 4309 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109805 4309 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109809 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109823 4309 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109829 4309 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109834 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109841 4309 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109848 4309 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109852 4309 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109857 4309 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109862 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109866 4309 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109870 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109875 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109879 4309 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:52.116823 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109884 4309 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109888 4309 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109892 4309 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109898 4309 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109902 4309 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109906 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109911 4309 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109918 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109923 4309 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109927 4309 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109932 4309 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109936 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109940 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109945 4309 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109949 4309 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109952 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109956 4309 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109961 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109965 4309 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:52.117292 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109969 4309 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109974 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109986 4309 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109991 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.109995 4309 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110135 4309 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110147 4309 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110156 4309 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110162 4309 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110168 4309 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110173 4309 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110180 4309 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110186 4309 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110191 4309 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110196 4309 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110202 4309 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110208 4309 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110213 4309 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110217 4309 flags.go:64] FLAG: --cgroup-root="" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110222 4309 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110230 4309 flags.go:64] FLAG: --client-ca-file="" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110235 4309 flags.go:64] FLAG: --cloud-provider="external" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110240 4309 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 21:57:52.117839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110251 4309 flags.go:64] FLAG: --cluster-domain="" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110256 4309 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110261 4309 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110266 4309 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110272 4309 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110278 4309 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110283 4309 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110288 4309 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110293 4309 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110298 4309 flags.go:64] FLAG: --contention-profiling="false" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110303 4309 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110308 4309 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110321 4309 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110326 4309 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110333 4309 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110339 4309 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110343 4309 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110348 4309 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110353 4309 flags.go:64] FLAG: --enable-server="true" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110358 4309 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110368 4309 flags.go:64] FLAG: --event-burst="100" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110373 4309 flags.go:64] FLAG: --event-qps="50" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110378 4309 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110383 4309 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 21:57:52.118388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110388 4309 flags.go:64] FLAG: --eviction-hard="" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110395 4309 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110399 4309 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110404 4309 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110409 4309 flags.go:64] FLAG: --eviction-soft="" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110417 4309 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110422 4309 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110427 4309 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110432 4309 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110436 4309 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110442 4309 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110446 4309 flags.go:64] FLAG: --feature-gates="" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110456 4309 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110461 4309 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110466 4309 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110472 4309 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110477 4309 flags.go:64] FLAG: --healthz-port="10248" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110482 4309 flags.go:64] FLAG: --help="false" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110486 4309 flags.go:64] FLAG: --hostname-override="ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110492 4309 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110496 4309 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110509 4309 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110516 4309 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 21:57:52.118998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110522 4309 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110526 4309 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110531 4309 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110536 4309 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110540 4309 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110545 4309 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110551 4309 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110555 4309 flags.go:64] FLAG: --kube-reserved="" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110560 4309 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110565 4309 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110570 4309 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110574 4309 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110579 4309 flags.go:64] FLAG: --lock-file="" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110583 4309 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110590 4309 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110596 4309 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110605 4309 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110610 4309 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110615 4309 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110620 4309 flags.go:64] FLAG: --logging-format="text" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110625 4309 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110630 4309 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110634 4309 flags.go:64] FLAG: --manifest-url="" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110639 4309 flags.go:64] FLAG: --manifest-url-header="" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110646 4309 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 21:57:52.119557 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110651 4309 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110658 4309 flags.go:64] FLAG: --max-pods="110" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110663 4309 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110668 4309 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110691 4309 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110696 4309 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110703 4309 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110708 4309 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110713 4309 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110724 4309 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110729 4309 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110734 4309 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110739 4309 flags.go:64] FLAG: --pod-cidr="" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110744 4309 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110749 4309 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110754 4309 flags.go:64] FLAG: --pods-per-core="0" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110759 4309 flags.go:64] FLAG: --port="10250" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110764 4309 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110769 4309 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-043b2de84a6d983eb" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110774 4309 flags.go:64] FLAG: --qos-reserved="" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110779 4309 flags.go:64] FLAG: --read-only-port="10255" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110786 4309 flags.go:64] FLAG: --register-node="true" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110791 4309 flags.go:64] FLAG: --register-with-taints="" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110797 4309 flags.go:64] FLAG: --registry-burst="10" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110802 4309 flags.go:64] FLAG: --registry-qps="5" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110807 4309 flags.go:64] FLAG: --reserved-cpus="" Nov 02 21:57:52.120177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110812 4309 flags.go:64] FLAG: --reserved-memory="" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110818 4309 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110823 4309 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110827 4309 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110832 4309 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110837 4309 flags.go:64] FLAG: --runonce="false" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110842 4309 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110848 4309 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110853 4309 flags.go:64] FLAG: --seccomp-default="false" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110858 4309 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110863 4309 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110868 4309 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110873 4309 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110879 4309 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110884 4309 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110889 4309 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110894 4309 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110898 4309 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110904 4309 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110908 4309 flags.go:64] FLAG: --system-cgroups="" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110913 4309 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110922 4309 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110926 4309 flags.go:64] FLAG: --tls-cert-file="" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110931 4309 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110938 4309 flags.go:64] FLAG: --tls-min-version="" Nov 02 21:57:52.120870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110943 4309 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110948 4309 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110955 4309 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110960 4309 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110965 4309 flags.go:64] FLAG: --v="2" Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110972 4309 flags.go:64] FLAG: --version="false" Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110979 4309 flags.go:64] FLAG: --vmodule="" Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110985 4309 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.110990 4309 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111229 4309 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111236 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111241 4309 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111245 4309 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111250 4309 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111255 4309 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111259 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111263 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111269 4309 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111273 4309 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111277 4309 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111282 4309 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:52.121489 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111288 4309 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111293 4309 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111297 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111302 4309 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111306 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111310 4309 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111314 4309 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111319 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111323 4309 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111328 4309 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111332 4309 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111337 4309 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111341 4309 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111348 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111352 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111357 4309 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111361 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111365 4309 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111369 4309 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111374 4309 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:52.122008 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111378 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111383 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111387 4309 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111391 4309 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111395 4309 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111400 4309 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111404 4309 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111409 4309 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111414 4309 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111418 4309 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111423 4309 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111427 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111432 4309 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111442 4309 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111447 4309 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111451 4309 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111455 4309 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111460 4309 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111464 4309 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111468 4309 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111473 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:52.122544 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111477 4309 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111482 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111486 4309 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111491 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111497 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111501 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111506 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111510 4309 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111517 4309 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111524 4309 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111529 4309 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111534 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111539 4309 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111545 4309 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111550 4309 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111554 4309 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111559 4309 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111563 4309 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111568 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:52.123143 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111572 4309 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111576 4309 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111582 4309 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111590 4309 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111595 4309 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111600 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111605 4309 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111610 4309 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111615 4309 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111620 4309 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111624 4309 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111628 4309 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111646 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111652 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111656 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111661 4309 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111665 4309 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111687 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111692 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:52.123652 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111697 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111701 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111705 4309 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111710 4309 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111714 4309 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111718 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111723 4309 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111727 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111731 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111735 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111740 4309 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111744 4309 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111749 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111753 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111757 4309 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:52.124252 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.111802 4309 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111858 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111866 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111871 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111877 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111882 4309 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111886 4309 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111891 4309 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111895 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111899 4309 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111904 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111908 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111912 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111916 4309 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111921 4309 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111925 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111930 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111934 4309 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111939 4309 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:52.124791 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111943 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111948 4309 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111952 4309 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111956 4309 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111961 4309 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111965 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111969 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.111998 4309 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112006 4309 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112011 4309 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112015 4309 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112019 4309 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112024 4309 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112028 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112032 4309 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112037 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112041 4309 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112046 4309 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112051 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:52.125327 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112056 4309 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112061 4309 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112065 4309 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112070 4309 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112074 4309 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112079 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112083 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112088 4309 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112092 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112097 4309 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112101 4309 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112105 4309 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112109 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112114 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112118 4309 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112122 4309 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112126 4309 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112131 4309 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112135 4309 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112140 4309 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:52.125876 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112144 4309 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112148 4309 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112152 4309 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112156 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112161 4309 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112166 4309 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112171 4309 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112175 4309 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112180 4309 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112184 4309 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112188 4309 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112192 4309 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112198 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112203 4309 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112208 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112212 4309 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112217 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112221 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112225 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112229 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:52.126375 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112234 4309 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112240 4309 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112247 4309 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112252 4309 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112257 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112261 4309 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112266 4309 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112270 4309 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112274 4309 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112278 4309 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112283 4309 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112287 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112292 4309 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112296 4309 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112302 4309 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112309 4309 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112314 4309 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112319 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112324 4309 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:52.126894 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112328 4309 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112333 4309 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112338 4309 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112342 4309 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112346 4309 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112350 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112355 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112360 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112366 4309 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.112370 4309 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.117533 4309 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.117554 4309 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.117571 4309 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.117576 4309 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117633 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117639 4309 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117643 4309 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117646 4309 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117649 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:52.127539 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117652 4309 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117655 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117658 4309 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117660 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117663 4309 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117666 4309 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117669 4309 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117685 4309 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117690 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117693 4309 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117696 4309 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117699 4309 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117702 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117704 4309 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117707 4309 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117710 4309 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117712 4309 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117715 4309 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117718 4309 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117720 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:52.128030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117723 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117726 4309 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117729 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117732 4309 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117735 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117738 4309 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117741 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117743 4309 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117746 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117748 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117751 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117755 4309 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117759 4309 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117762 4309 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117765 4309 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117768 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117770 4309 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117773 4309 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117776 4309 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:52.128519 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117778 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117781 4309 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117784 4309 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117787 4309 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117789 4309 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117792 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117795 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117798 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117800 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117803 4309 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117805 4309 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117808 4309 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117810 4309 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117813 4309 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117815 4309 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117818 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117821 4309 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117823 4309 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117826 4309 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117829 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:52.129193 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117831 4309 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117834 4309 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117837 4309 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117839 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117842 4309 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117845 4309 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117847 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117850 4309 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117853 4309 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117856 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117859 4309 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117861 4309 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117864 4309 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117868 4309 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117871 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117875 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117878 4309 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117881 4309 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117884 4309 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:52.129790 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117886 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117889 4309 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117891 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117894 4309 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117896 4309 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117899 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117902 4309 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117904 4309 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117907 4309 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117909 4309 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117912 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117914 4309 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117917 4309 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117919 4309 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117922 4309 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117924 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117927 4309 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117929 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117932 4309 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:52.130395 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117935 4309 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117937 4309 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117939 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.117942 4309 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.117972 4309 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118110 4309 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118116 4309 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118119 4309 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118122 4309 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118125 4309 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118128 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118131 4309 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118133 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118136 4309 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118138 4309 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118141 4309 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:52.131170 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118144 4309 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118146 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118149 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118152 4309 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118155 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118157 4309 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118160 4309 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118162 4309 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118165 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118168 4309 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118171 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118173 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118176 4309 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118179 4309 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118181 4309 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118184 4309 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118186 4309 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118189 4309 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118192 4309 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118194 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:52.131712 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118197 4309 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118200 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118202 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118205 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118208 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118210 4309 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118213 4309 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118216 4309 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118219 4309 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118221 4309 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118224 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118226 4309 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118229 4309 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118233 4309 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118237 4309 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118240 4309 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118243 4309 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118246 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118249 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:52.132252 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118251 4309 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118254 4309 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118256 4309 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118259 4309 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118261 4309 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118264 4309 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118267 4309 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118270 4309 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118272 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118278 4309 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118280 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118283 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118286 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118288 4309 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118291 4309 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118293 4309 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118296 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118298 4309 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118301 4309 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118303 4309 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:52.132997 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118306 4309 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118309 4309 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118312 4309 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118314 4309 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118317 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118320 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118322 4309 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118325 4309 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118328 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118330 4309 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118333 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118335 4309 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118338 4309 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118340 4309 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118343 4309 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118346 4309 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118348 4309 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118351 4309 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118353 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118356 4309 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:52.133518 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118359 4309 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118363 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118366 4309 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118368 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118371 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118375 4309 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118379 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118382 4309 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118385 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118388 4309 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118391 4309 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118393 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118396 4309 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118399 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118402 4309 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:52.134030 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118405 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.118428 4309 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118477 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118482 4309 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118485 4309 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118488 4309 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118491 4309 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118494 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118497 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118500 4309 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118502 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118505 4309 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118508 4309 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118510 4309 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118513 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118516 4309 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 21:57:52.134463 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118519 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118522 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118524 4309 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118528 4309 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118531 4309 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118533 4309 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118536 4309 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118539 4309 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118542 4309 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118544 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118547 4309 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118549 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118552 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118555 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118557 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118560 4309 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118563 4309 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118565 4309 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118568 4309 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118571 4309 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 21:57:52.134895 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118573 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118576 4309 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118578 4309 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118582 4309 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118586 4309 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118590 4309 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118593 4309 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118595 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118598 4309 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118601 4309 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118603 4309 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118606 4309 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118609 4309 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118611 4309 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118616 4309 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118620 4309 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118623 4309 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118626 4309 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118628 4309 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 21:57:52.135425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118631 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118634 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118636 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118639 4309 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118641 4309 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118644 4309 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118646 4309 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118649 4309 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118651 4309 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118654 4309 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118657 4309 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118660 4309 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118662 4309 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118665 4309 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118667 4309 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118670 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118687 4309 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118690 4309 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118693 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118695 4309 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 21:57:52.136061 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118698 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118701 4309 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118704 4309 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118706 4309 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118709 4309 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118712 4309 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118714 4309 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118718 4309 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118722 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118725 4309 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118728 4309 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118730 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118733 4309 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118735 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118738 4309 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118742 4309 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118746 4309 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118749 4309 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118752 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 21:57:52.136546 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118754 4309 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118757 4309 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118760 4309 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118763 4309 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118769 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118772 4309 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118774 4309 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118777 4309 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118779 4309 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118782 4309 feature_gate.go:386] unrecognized feature gate: Example Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118785 4309 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118787 4309 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118790 4309 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:52.118792 4309 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.118844 4309 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.119907 4309 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.119962 4309 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.120139 4309 server.go:1014] "Starting client certificate rotation" Nov 02 21:57:52.137033 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.120274 4309 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 21:57:52.137469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.120330 4309 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 21:52:12 +0000 UTC" deadline="2028-04-08 23:28:18.400813034 +0000 UTC" Nov 02 21:57:52.137469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.120363 4309 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12553h30m26.280451479s" Nov 02 21:57:52.137469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.120591 4309 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:57:52.137469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.121623 4309 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 21:57:52.137469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.122971 4309 log.go:25] "Validated CRI v1 runtime API" Nov 02 21:57:52.137469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.124370 4309 log.go:25] "Validated CRI v1 image API" Nov 02 21:57:52.137469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.125165 4309 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 21:57:52.137469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.126454 4309 fs.go:135] Filesystem UUIDs: map[3f301e84-194d-4e77-919f-9a6b3385d3f0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 da48cd05-0025-45b6-b526-edb7a6de925a:/dev/nvme0n1p3] Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.126474 4309 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/057d7893582ccea742641ed31efdd53b0566dc7cf383a96c59161bea63f80c44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/057d7893582ccea742641ed31efdd53b0566dc7cf383a96c59161bea63f80c44/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07389bee425d03feac29195ca6f0778dd11eef65839eddc74e9b6ace5fa2ec0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07389bee425d03feac29195ca6f0778dd11eef65839eddc74e9b6ace5fa2ec0d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fcecc27505fe39fd7d70ff9265661b66b4a61bcdf534e13e2fc9200b8450041/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fcecc27505fe39fd7d70ff9265661b66b4a61bcdf534e13e2fc9200b8450041/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15532698420d72d2aafd4a0b6f08d3bab05526ece36c670e8dfc3bad6565a8a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15532698420d72d2aafd4a0b6f08d3bab05526ece36c670e8dfc3bad6565a8a5/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f82fb253d3e5ce43f20e9f3437d710ed427d55dde183183327a8196762bbdfe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f82fb253d3e5ce43f20e9f3437d710ed427d55dde183183327a8196762bbdfe/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a4b3104ff10beff33d55a15173ed460e1cffc6728a6f7dfa19bc300adf4bd7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a4b3104ff10beff33d55a15173ed460e1cffc6728a6f7dfa19bc300adf4bd7d/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9486167f074d3846cd429fd96ca030d5e09c311bdb24e08c58d0d51fab754a3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9486167f074d3846cd429fd96ca030d5e09c311bdb24e08c58d0d51fab754a3e/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d026b8e0f166199936add29edb8a6e8e091375f4b0bbcea6c2fdcecb1f415ee5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d026b8e0f166199936add29edb8a6e8e091375f4b0bbcea6c2fdcecb1f415ee5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e48fe2500e9a7aeaf9c73f1b49aaa3991ba80ca36e76be503ef8c40d8fdf1642/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e48fe2500e9a7aeaf9c73f1b49aaa3991ba80ca36e76be503ef8c40d8fdf1642/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f172b60e0b85874cb3ddda149bd16d5149c21eaef57032b0da728586f2b97831/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f172b60e0b85874cb3ddda149bd16d5149c21eaef57032b0da728586f2b97831/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f32627a3012229ad56a0b727ad7cd27cd93702777ef1e483665ca47e38333f22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f32627a3012229ad56a0b727ad7cd27cd93702777ef1e483665ca47e38333f22/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4e317226fd6943a30f9538578bab99ade5114f94aa39e42bb043d3b8df65908/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4e317226fd6943a30f9538578bab99ade5114f94aa39e42bb043d3b8df65908/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10158047-423e-4241-b46d-6ad29dece441/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/10158047-423e-4241-b46d-6ad29dece441/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ee5c83e-ba60-41ef-a638-a77cdff3c824/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3ee5c83e-ba60-41ef-a638-a77cdff3c824/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/527b2ac7-739e-4f53-b553-474cad1eb712/volumes/kubernetes.io~projected/kube-api-access-6rqjx:{mountpoint:/var/lib/kubelet/pods/527b2ac7-739e-4f53-b553-474cad1eb712/volumes/kubernetes.io~projected/kube-api-access-6rqjx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/53101ce6-458d-4b8f-ac21-36679d929ba0/volumes/kubernetes.io~projected/kube-api-access-9j57h:{mountpoint:/var/lib/kubelet/pods/53101ce6-458d-4b8f-ac21-36679d929ba0/volumes/kubernetes.io~projected/kube-api-access-9j57h major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a/volumes/kubernetes.io~projected/kube-api-access-nccm2:{mountpoint:/var/lib/kubelet/pods/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a/volumes/kubernetes.io~projected/kube-api-access-nccm2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67af9038-04a3-4574-96fd-23a3bc52d944/volumes/kubernetes.io~projected/kube-api-access-vfc7z:{mountpoint:/var/lib/kubelet/pods/67af9038-04a3-4574-96fd-23a3bc52d944/volumes/kubernetes.io~projected/kube-api-access-vfc7z major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/735421c1-678c-4cc4-a07e-57ffb0e9ca29/volumes/kubernetes.io~projected/kube-api-access-wtsdx:{mountpoint:/var/lib/kubelet/pods/735421c1-678c-4cc4-a07e-57ffb0e9ca29/volumes/kubernetes.io~projected/kube-api-access-wtsdx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79214604-184d-4eb5-b63e-d2a30209835c/volumes/kubernetes.io~projected/kube-api-access-mqfz5:{mountpoint:/var/lib/kubelet/pods/79214604-184d-4eb5-b63e-d2a30209835c/volumes/kubernetes.io~projected/kube-api-access-mqfz5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89a526ea-68b3-40e3-a474-02fa8cec0ab0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/89a526ea-68b3-40e3-a474-02fa8cec0ab0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89a526ea-68b3-40e3-a474-02fa8cec0ab0/volumes/kubernetes.io~projected/kube-api-access-bdfwb:{mountpoint:/var/lib/kubelet/pods/89a526ea-68b3-40e3-a474-02fa8cec0ab0/volumes/kubernetes.io~projected/kube-api-access-bdfwb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89a526ea-68b3-40e3-a474-02fa8cec0ab0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/89a526ea-68b3-40e3-a474-02fa8cec0ab0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92283fa1-fe9a-4287-a313-176acb992dc9/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/92283fa1-fe9a-4287-a313-176acb992dc9/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92283fa1-fe9a-4287-a313-176acb992dc9/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/92283fa1-fe9a-4287-a313-176acb992dc9/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92283fa1-fe9a-4287-a313-176acb992dc9/volumes/kubernetes.io~projected/kube-api-access-dbh7g:{mountpoint:/var/lib/kubelet/pods/92283fa1-fe9a-4287-a313-176acb992dc9/volumes/kubernetes.io~projected/kube-api-access-dbh7g major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a18dfddd-46cd-48c6-898c-0b313c829499/volumes/kubernetes.io~projected/kube-api-access-dtt6c:{mountpoint:/var/lib/kubelet/pods/a18dfddd-46cd-48c6-898c-0b313c829499/volumes/kubernetes.io~projected/kube-api-access-dtt6c major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5023155-869d-44af-9168-7a44f3cdc1a3/volumes/kubernetes.io~projected/kube-api-access-pvjzw:{mountpoint:/var/lib/kubelet/pods/c5023155-869d-44af-9168-7a44f3cdc1a3/volumes/kubernetes.io~projected/kube-api-access-pvjzw major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6d226244c498f7b532883ed123c8d6c22749acdd7e48832176645e5a8644949d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/049ab9ee73eca63e01d96a544566c001aafad226f2e6a0bf28283d3223094377/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0d0fba069f879e48012e591bde06a06bb203cdc54e4c1c18c2f351b6465784a7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/168be06afe87b146c0f533a5ba64498371383f0f335719796b3b69fad54b114b/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2b17bfa6c028f5c743bc4e1f92fa2a8cf8faaa14ac922e3a9fe5e470c73efa71/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8c84cdd231984574f46e84026ce941036c7e36840227553dbcbba4cb544bb476/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d4fffb8faabf41f19fcb5e21e95bfd13c9323d8fcc710f1afe8a301a00e63739/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7eb98ef2cb3ce92e70636d4a77933f84317582974724f8b0640604f6f3560cc1/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ce7391ce08b083ae1d12fc97813b5186880547f879205b5d473b05279c9b7620/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/223c1e819e810af322ae1a987db263f3a5acfbfbe0a81860c85951ed9e29d87b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/8ab5fdac44ae7baab7bb54e8250d8e84a6fda7e297b24df882bd6d63295c4b61/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/841acb6a58a8d5e1591a3cf4d080f9bb2e976bfbf9211cd818a2bf242a7a4d99/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/8d5cd24d35db2c8d2c5817fe9d5c7d8ffa539f31290ee5e089707392afcfacda/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/fb0d95c789192a2d960c3ca49855af8dd4ef6f688c96938b2259f663af66849d/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9b5166d57b19eac805e8f044c63fc7c187038ca06f038713a813b5af75da269c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a226cceaed1e6fd27bf1a0c36da64d1ae73e7a9652695143ed704ed00732a9e4/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/27b9ea4b66a680478710511c61e052752fb25d7178d66fb7f1e1782502e11523/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/230d591418464507a209e469bb487b9e6b3c99badd2f50fec8ea4d66fa1aa488/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f0ef0beba1cdc5aa6d040f945fcd81b6109fe75756d3bace82cfad872c1171f8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7b4a636427496bb13ebb4460e48665f6f32956fbccab1560374aaa2d808267e8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6ae9f1a13ebf35ff1fb401d0cb84dcaba14b5037ab756a5c809d834062a60a40/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/7e0ead74c9aa462048272f6e852ef53d9c73305cebc25373baa7069700579e2a/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/9a07f784e7e20dc0f56a19f5a53981f3550194e7e5baa1a6d81333b72d044329/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/e2a524d8ec8cf73a0a4a6942a9209d7020d963e78a5b0ecfded54aa523f42c49/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/a609cdaed13ce912a10b6e47fa525b9f9272395d669dadb4bd24471ba502838f/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/94946cf3148831b4d6f6d8e3e81c0bf9b98ebb275e197dfaf14dc3ae5139ec65/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a84ac24f031352ed3376af7b552cd487fe2a822b5da0801ff0546fbcdf072f4d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b48d1ee1d0c8fd929e694d610e21b71d11d87ec4f63bfa637d690af793d3ffa1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6e5ce9934bde91d1a14cfce3473c3490e80b44c5a8849a9924b9aff1fe67770b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/28d50f5e178bdcde6c330f51a99b2cb804176a7523f5f561149c95552a25cdbe/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/537a3f8d6ca637a7e6b2db59ae724972b979be792af0b5f22dd486abb1da2b00/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/7a5b637f29744831b746ba7f8bc7e51601e566d98ec42e060192b1d7c4ddb1bb/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d5ac4ed020b2756ccc0f68e69a066a54ebb207a0111673b1c5ce324eb863a2f0/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b0ac575f036a1190e3c6a0f13aa8ac3ea66071f586983846fefb87c4a4242872/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f383193c51ad23a6d63fce5193bd0e9b82675b626fa345a6f72d42adfa9347a3/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.133622 4309 manager.go:217] Machine: {Timestamp:2026-11-02 21:57:52.133170097 +0000 UTC m=+0.071283306 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108776 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23a626a4d5885ce155f7e72b857a90 SystemUUID:ec23a626-a4d5-885c-e155-f7e72b857a90 BootID:558fdbcc-ee6f-4202-994d-f73b85571922 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/527b2ac7-739e-4f53-b553-474cad1eb712/volumes/kubernetes.io~projected/kube-api-access-6rqjx DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4e317226fd6943a30f9538578bab99ade5114f94aa39e42bb043d3b8df65908/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67af9038-04a3-4574-96fd-23a3bc52d944/volumes/kubernetes.io~projected/kube-api-access-vfc7z DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a18dfddd-46cd-48c6-898c-0b313c829499/volumes/kubernetes.io~projected/kube-api-access-dtt6c DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f82fb253d3e5ce43f20e9f3437d710ed427d55dde183183327a8196762bbdfe/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/92283fa1-fe9a-4287-a313-176acb992dc9/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/53101ce6-458d-4b8f-ac21-36679d929ba0/volumes/kubernetes.io~projected/kube-api-access-9j57h DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ee5c83e-ba60-41ef-a638-a77cdff3c824/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/92283fa1-fe9a-4287-a313-176acb992dc9/volumes/kubernetes.io~projected/kube-api-access-dbh7g DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/057d7893582ccea742641ed31efdd53b0566dc7cf383a96c59161bea63f80c44/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89a526ea-68b3-40e3-a474-02fa8cec0ab0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f172b60e0b85874cb3ddda149bd16d5149c21eaef57032b0da728586f2b97831/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79214604-184d-4eb5-b63e-d2a30209835c/volumes/kubernetes.io~projected/kube-api-access-mqfz5 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89a526ea-68b3-40e3-a474-02fa8cec0ab0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d026b8e0f166199936add29edb8a6e8e091375f4b0bbcea6c2fdcecb1f415ee5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9486167f074d3846cd429fd96ca030d5e09c311bdb24e08c58d0d51fab754a3e/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07389bee425d03feac29195ca6f0778dd11eef65839eddc74e9b6ace5fa2ec0d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/89a526ea-68b3-40e3-a474-02fa8cec0ab0/volumes/kubernetes.io~projected/kube-api-access-bdfwb DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a/volumes/kubernetes.io~projected/kube-api-access-nccm2 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fcecc27505fe39fd7d70ff9265661b66b4a61bcdf534e13e2fc9200b8450041/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5023155-869d-44af-9168-7a44f3cdc1a3/volumes/kubernetes.io~projected/kube-api-access-pvjzw DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/735421c1-678c-4cc4-a07e-57ffb0e9ca29/volumes/kubernetes.io~projected/kube-api-access-wtsdx DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15532698420d72d2aafd4a0b6f08d3bab05526ece36c670e8dfc3bad6565a8a5/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/10158047-423e-4241-b46d-6ad29dece441/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/92283fa1-fe9a-4287-a313-176acb992dc9/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a4b3104ff10beff33d55a15173ed460e1cffc6728a6f7dfa19bc300adf4bd7d/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e48fe2500e9a7aeaf9c73f1b49aaa3991ba80ca36e76be503ef8c40d8fdf1642/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f32627a3012229ad56a0b727ad7cd27cd93702777ef1e483665ca47e38333f22/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:15532698420d72d MacAddress:66:e9:ea:fa:b8:1d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dd:d9:b2:9f Speed:0 Mtu:9001} {Name:br-int MacAddress:56:71:5f:c4:cd:09 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:dd:d9:b2:9f Speed:0 Mtu:9001} {Name:f32627a3012229a MacAddress:be:80:b7:d4:7b:c6 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:16:60:3a:84:1d:bf Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:0b:7f:89:65:2c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.133867 4309 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.133909 4309 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134166 4309 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134186 4309 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-9.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134302 4309 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134309 4309 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134321 4309 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134334 4309 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134415 4309 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134561 4309 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134689 4309 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134722 4309 kubelet.go:493] "Attempting to sync node with API server" Nov 02 21:57:52.137777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134733 4309 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134742 4309 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134750 4309 kubelet.go:405] "Adding apiserver pod source" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.134758 4309 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.135323 4309 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.135335 4309 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.135772 4309 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.137557 4309 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.137567 4309 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.137666 4309 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.137882 4309 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.137909 4309 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138014 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138029 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138039 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138047 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138052 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138058 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138063 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138069 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138076 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138082 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138096 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138106 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138154 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 21:57:52.138389 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.138160 4309 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 21:57:52.141573 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.141555 4309 server.go:1263] "Started kubelet" Nov 02 21:57:52.141687 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.141604 4309 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 21:57:52.141803 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.141751 4309 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 21:57:52.141864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.141855 4309 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 21:57:52.142281 ip-10-0-136-9 systemd[1]: Started Kubernetes Kubelet. Nov 02 21:57:52.142509 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.142308 4309 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 21:57:52.147322 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.147021 4309 server.go:317] "Adding debug handlers to kubelet server" Nov 02 21:57:52.148126 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.148103 4309 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 21:57:52.148482 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.148313 4309 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 21:57:52.148482 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.148409 4309 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 21:57:52.148482 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.148425 4309 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 21:57:52.148702 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.148526 4309 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 21:57:52.148702 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.148557 4309 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 21:52:14 +0000 UTC" deadline="2028-07-04 18:18:45.422754045 +0000 UTC" Nov 02 21:57:52.148702 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.148594 4309 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14636h20m53.274164854s" Nov 02 21:57:52.148865 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:57:52.148804 4309 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-9.us-east-2.compute.internal\" not found" Nov 02 21:57:52.150142 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150103 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a18dfddd-46cd-48c6-898c-0b313c829499" volumeName="kubernetes.io/configmap/a18dfddd-46cd-48c6-898c-0b313c829499-iptables-alerter-script" seLinuxMountContext="" Nov 02 21:57:52.150142 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150137 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a18dfddd-46cd-48c6-898c-0b313c829499" volumeName="kubernetes.io/projected/a18dfddd-46cd-48c6-898c-0b313c829499-kube-api-access-dtt6c" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150155 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="527b2ac7-739e-4f53-b553-474cad1eb712" volumeName="kubernetes.io/configmap/527b2ac7-739e-4f53-b553-474cad1eb712-serviceca" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150168 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="527b2ac7-739e-4f53-b553-474cad1eb712" volumeName="kubernetes.io/projected/527b2ac7-739e-4f53-b553-474cad1eb712-kube-api-access-6rqjx" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150180 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53101ce6-458d-4b8f-ac21-36679d929ba0" volumeName="kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150191 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53101ce6-458d-4b8f-ac21-36679d929ba0" volumeName="kubernetes.io/projected/53101ce6-458d-4b8f-ac21-36679d929ba0-kube-api-access-9j57h" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150202 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="735421c1-678c-4cc4-a07e-57ffb0e9ca29" volumeName="kubernetes.io/projected/735421c1-678c-4cc4-a07e-57ffb0e9ca29-kube-api-access-wtsdx" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150214 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79214604-184d-4eb5-b63e-d2a30209835c" volumeName="kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-cni-binary-copy" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150229 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89a526ea-68b3-40e3-a474-02fa8cec0ab0" volumeName="kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-env-overrides" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150240 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89a526ea-68b3-40e3-a474-02fa8cec0ab0" volumeName="kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-script-lib" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150251 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53101ce6-458d-4b8f-ac21-36679d929ba0" volumeName="kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150262 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67af9038-04a3-4574-96fd-23a3bc52d944" volumeName="kubernetes.io/projected/67af9038-04a3-4574-96fd-23a3bc52d944-kube-api-access-vfc7z" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150273 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79214604-184d-4eb5-b63e-d2a30209835c" volumeName="kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-multus-daemon-config" seLinuxMountContext="" Nov 02 21:57:52.150288 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150284 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89a526ea-68b3-40e3-a474-02fa8cec0ab0" volumeName="kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-config" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150300 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89a526ea-68b3-40e3-a474-02fa8cec0ab0" volumeName="kubernetes.io/projected/89a526ea-68b3-40e3-a474-02fa8cec0ab0-kube-api-access-bdfwb" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150312 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92283fa1-fe9a-4287-a313-176acb992dc9" volumeName="kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-tmp" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150323 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10158047-423e-4241-b46d-6ad29dece441" volumeName="kubernetes.io/secret/10158047-423e-4241-b46d-6ad29dece441-agent-certs" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150335 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="53101ce6-458d-4b8f-ac21-36679d929ba0" volumeName="kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-binary-copy" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150346 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a" volumeName="kubernetes.io/projected/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-kube-api-access-nccm2" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150357 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79214604-184d-4eb5-b63e-d2a30209835c" volumeName="kubernetes.io/projected/79214604-184d-4eb5-b63e-d2a30209835c-kube-api-access-mqfz5" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150368 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5023155-869d-44af-9168-7a44f3cdc1a3" volumeName="kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150379 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10158047-423e-4241-b46d-6ad29dece441" volumeName="kubernetes.io/configmap/10158047-423e-4241-b46d-6ad29dece441-konnectivity-ca" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150390 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ee5c83e-ba60-41ef-a638-a77cdff3c824" volumeName="kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150402 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="735421c1-678c-4cc4-a07e-57ffb0e9ca29" volumeName="kubernetes.io/empty-dir/735421c1-678c-4cc4-a07e-57ffb0e9ca29-tmp-dir" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150413 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89a526ea-68b3-40e3-a474-02fa8cec0ab0" volumeName="kubernetes.io/secret/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150451 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92283fa1-fe9a-4287-a313-176acb992dc9" volumeName="kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-etc-tuned" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150467 4309 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92283fa1-fe9a-4287-a313-176acb992dc9" volumeName="kubernetes.io/projected/92283fa1-fe9a-4287-a313-176acb992dc9-kube-api-access-dbh7g" seLinuxMountContext="" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150479 4309 reconstruct.go:97] "Volume reconstruction finished" Nov 02 21:57:52.150864 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150487 4309 reconciler.go:29] "Reconciler: start to sync state" Nov 02 21:57:52.151656 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150939 4309 factory.go:55] Registering systemd factory Nov 02 21:57:52.151656 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.150954 4309 factory.go:223] Registration of the systemd container factory successfully Nov 02 21:57:52.151656 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.151374 4309 factory.go:153] Registering CRI-O factory Nov 02 21:57:52.151656 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.151389 4309 factory.go:223] Registration of the crio container factory successfully Nov 02 21:57:52.151656 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.151439 4309 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 21:57:52.151656 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.151458 4309 factory.go:103] Registering Raw factory Nov 02 21:57:52.151656 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.151471 4309 manager.go:1199] Started watching for new ooms in manager Nov 02 21:57:52.152043 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.151790 4309 manager.go:319] Starting recovery of all containers Nov 02 21:57:52.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.153838 4309 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:52.155535 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.155461 4309 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:52.155958 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.155936 4309 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:52.156892 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.156864 4309 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 21:57:52.158590 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.158345 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 21:57:52.159709 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.159446 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 21:57:52.178994 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.178959 4309 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 21:57:52.180178 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.180154 4309 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 21:57:52.180287 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.180182 4309 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 21:57:52.180287 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.180205 4309 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 21:57:52.180287 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:57:52.180246 4309 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 21:57:52.183616 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.183592 4309 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 21:57:52.187441 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.187420 4309 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="b2f2e5bc78b55ed667e333c9d6bed91cecb10569f36aeb418617002544acdfc0" exitCode=0 Nov 02 21:57:52.187517 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.187441 4309 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="a04b6307e098ad81ad63742a71d0061fd615648316f833dd412b037d2d63697b" exitCode=0 Nov 02 21:57:52.187517 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.187453 4309 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="1a885ed94fb6238b4b69ab3750ae929fb46d9d6c63ae3cc219f47f859764b78a" exitCode=0 Nov 02 21:57:52.187517 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.187464 4309 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="7062bc8a5e7d839f7df33c0d48945671721ade022ea0735011e68ed835a72f2b" exitCode=0 Nov 02 21:57:52.187517 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.187471 4309 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="debdfc9ca4c3a790b88f6ee1965a5885a1197a020b4080fa917e5754f1c0e459" exitCode=0 Nov 02 21:57:52.187517 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.187478 4309 generic.go:356] "Generic (PLEG): container finished" podID="53101ce6-458d-4b8f-ac21-36679d929ba0" containerID="4938cd3b82a6b6f280ffa365de8be87aa9d63aab5e374a49817a9373dfc9ee55" exitCode=0 Nov 02 21:57:52.197936 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.197913 4309 generic.go:356] "Generic (PLEG): container finished" podID="22d5d23fd92916dc8b8ecfbd0ec29fad" containerID="511949b91d25abc3ed0673e142b690d4197ebd732ea404b1afe5f3e1d3b6735f" exitCode=0 Nov 02 21:57:52.206169 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.206151 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 21:57:52.206531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.206511 4309 generic.go:356] "Generic (PLEG): container finished" podID="89a526ea-68b3-40e3-a474-02fa8cec0ab0" containerID="0c2690e17dc40dc49f1100d2bfb3f95405598913759a45096722f0e0505532ab" exitCode=1 Nov 02 21:57:52.213373 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.213352 4309 manager.go:324] Recovery completed Nov 02 21:57:52.226790 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.226770 4309 cpu_manager.go:226] "Starting" policy="none" Nov 02 21:57:52.226790 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.226786 4309 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 21:57:52.226915 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.226805 4309 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 21:57:52.226956 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.226943 4309 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 21:57:52.226990 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.226961 4309 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 21:57:52.226990 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.226982 4309 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 21:57:52.227048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.226989 4309 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 21:57:52.227048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.226997 4309 policy_none.go:50] "Start" Nov 02 21:57:52.227048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.227008 4309 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 21:57:52.227048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.227020 4309 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 21:57:52.232171 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.232157 4309 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 21:57:52.232220 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.232173 4309 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 21:57:52.232220 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.232179 4309 policy_none.go:44] "Start" Nov 02 21:57:52.236020 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.235989 4309 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 21:57:52.238622 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.238602 4309 manager.go:341] "Starting Device Plugin manager" Nov 02 21:57:52.239128 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:57:52.238627 4309 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 21:57:52.239182 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.239131 4309 server.go:87] "Starting device plugin registration server" Nov 02 21:57:52.239390 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.239379 4309 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 21:57:52.239435 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.239393 4309 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 21:57:52.239545 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.239520 4309 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 21:57:52.239700 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.239639 4309 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 21:57:52.239759 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.239722 4309 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 21:57:52.239759 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.239733 4309 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 21:57:52.239876 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.239801 4309 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T21:57:52.239697712Z","UUID":"b64c12ed-4186-4caf-b095-96434d9f5b37","Handler":null,"Name":"","Endpoint":""} Nov 02 21:57:52.240939 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:57:52.240913 4309 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 21:57:52.241167 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.241146 4309 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 21:57:52.241231 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.241178 4309 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 21:57:52.280428 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.280351 4309 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal"] Nov 02 21:57:52.280846 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.280789 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"22d5d23fd92916dc8b8ecfbd0ec29fad","Type":"ContainerStarted","Data":"ca389cc7a2acb725db713497cfd983cae2ee99e96473fba7d5a94432a356ccee"} Nov 02 21:57:52.280952 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.280858 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"22d5d23fd92916dc8b8ecfbd0ec29fad","Type":"ContainerDied","Data":"511949b91d25abc3ed0673e142b690d4197ebd732ea404b1afe5f3e1d3b6735f"} Nov 02 21:57:52.280952 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.280881 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"22d5d23fd92916dc8b8ecfbd0ec29fad","Type":"ContainerStarted","Data":"4f82fb253d3e5ce43f20e9f3437d710ed427d55dde183183327a8196762bbdfe"} Nov 02 21:57:52.280952 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.280900 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"ad3b40d52cd903e26b09603b8260563a","Type":"ContainerStarted","Data":"1166cfd8fcd241c6ad2e73f9205950b8eef83d6392b545e87b2103d0f0239b22"} Nov 02 21:57:52.280952 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.280920 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" event={"ID":"ad3b40d52cd903e26b09603b8260563a","Type":"ContainerStarted","Data":"f172b60e0b85874cb3ddda149bd16d5149c21eaef57032b0da728586f2b97831"} Nov 02 21:57:52.281175 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.280969 4309 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.281175 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.281057 4309 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.291793 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.291777 4309 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:57:52.291850 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:57:52.291833 4309 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.292303 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.292289 4309 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:57:52.292385 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:57:52.292328 4309 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.340441 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.340419 4309 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 21:57:52.341664 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.341643 4309 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 21:57:52.341777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.341699 4309 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 21:57:52.341777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.341712 4309 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-9.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 02 21:57:52.341777 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.341742 4309 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.353440 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.353422 4309 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.353499 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.353481 4309 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.358076 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.358056 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad3b40d52cd903e26b09603b8260563a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"ad3b40d52cd903e26b09603b8260563a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.358158 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.358083 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.358158 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.358098 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.459166 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.459126 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad3b40d52cd903e26b09603b8260563a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"ad3b40d52cd903e26b09603b8260563a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.459166 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.459163 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.459398 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.459182 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.459398 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.459250 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ad3b40d52cd903e26b09603b8260563a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"ad3b40d52cd903e26b09603b8260563a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.459398 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.459301 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:52.459398 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:52.459344 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d5d23fd92916dc8b8ecfbd0ec29fad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" (UID: \"22d5d23fd92916dc8b8ecfbd0ec29fad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:53.135599 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.135561 4309 apiserver.go:52] "Watching apiserver" Nov 02 21:57:53.144843 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.144818 4309 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 21:57:53.145182 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.145165 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-jg7pl","openshift-multus/multus-dg6dc","kube-system/konnectivity-agent-zxdmr","kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-6jc58","openshift-multus/network-metrics-daemon-4l48q","kube-system/global-pull-secret-syncer-qkl72","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst","openshift-network-diagnostics/network-check-target-pft59","openshift-network-operator/iptables-alerter-lqhnn","openshift-ovn-kubernetes/ovnkube-node-pgrkb","openshift-cluster-node-tuning-operator/tuned-rszqx","openshift-dns/node-resolver-kclzj"] Nov 02 21:57:53.148176 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.148156 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:53.149670 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.149653 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 21:57:53.149841 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.149827 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:57:53.150103 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.150081 4309 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 21:57:53.150308 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.150293 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 21:57:53.151196 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.151025 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 21:57:53.151350 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.151335 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 21:57:53.151417 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.151394 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n7kkn\"" Nov 02 21:57:53.151613 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.151593 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gf8xp\"" Nov 02 21:57:53.151845 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.151830 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 21:57:53.151912 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.151879 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 21:57:53.152102 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.152085 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 21:57:53.152230 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.152212 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9dzjg\"" Nov 02 21:57:53.152282 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.152226 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 21:57:53.152333 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.152317 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 21:57:53.152525 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.152497 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 21:57:53.152590 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.152552 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6t4jd\"" Nov 02 21:57:53.152638 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.152601 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 21:57:53.153127 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.153103 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 21:57:53.153350 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.153324 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 21:57:53.153443 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.153427 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 21:57:53.153630 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.153616 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sd22g\"" Nov 02 21:57:53.153730 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.153717 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 21:57:53.153872 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.153847 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7q497\"" Nov 02 21:57:53.153935 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.153718 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5tk4z\"" Nov 02 21:57:53.153992 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.153977 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 21:57:53.154121 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.154105 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 21:57:53.154176 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.154149 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dx8bh\"" Nov 02 21:57:53.154270 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.154252 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 21:57:53.154431 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.154412 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 21:57:53.154488 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.154456 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 21:57:53.154614 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.154600 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-447sr\"" Nov 02 21:57:53.154912 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.154889 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vjf48\"" Nov 02 21:57:53.154979 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.154960 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 21:57:53.155189 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155169 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4qcf6\"" Nov 02 21:57:53.155297 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155223 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 21:57:53.155349 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155288 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 21:57:53.155394 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155365 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 21:57:53.155445 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155406 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 21:57:53.155494 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155454 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 21:57:53.155494 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155458 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wtxk8\"" Nov 02 21:57:53.155565 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155502 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 21:57:53.155908 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155892 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 21:57:53.155946 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.155897 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 21:57:53.156056 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.156045 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 21:57:53.162434 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162418 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/527b2ac7-739e-4f53-b553-474cad1eb712-kube-api-access-6rqjx\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:53.162480 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162446 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-k8s-cni-cncf-io\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.162480 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162463 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.162546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162483 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-lib-modules\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.162578 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162546 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-etc-tuned\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.162608 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162595 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a18dfddd-46cd-48c6-898c-0b313c829499-iptables-alerter-script\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:53.162636 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162625 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsdx\" (UniqueName: \"kubernetes.io/projected/735421c1-678c-4cc4-a07e-57ffb0e9ca29-kube-api-access-wtsdx\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:53.162669 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162650 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-system-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.162737 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162667 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-etc-tuned\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.162737 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162687 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10158047-423e-4241-b46d-6ad29dece441-agent-certs\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:53.162737 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162712 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-netns\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.162871 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162858 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovn-node-metrics-cert\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.162913 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162881 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-dbus\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:53.162913 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162897 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-kubelet-config\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:53.162913 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162904 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a18dfddd-46cd-48c6-898c-0b313c829499-iptables-alerter-script\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:53.163029 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162918 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.163029 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162960 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-multus-daemon-config\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.163029 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162969 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10158047-423e-4241-b46d-6ad29dece441-agent-certs\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:53.163029 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.162981 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.163029 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163000 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.163221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163065 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-env-overrides\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.163221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163186 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-binary-copy\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.163221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163192 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-var-lib-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.163221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163206 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovn-node-metrics-cert\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.163355 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163224 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-env-overrides\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.163355 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163222 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.163355 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163231 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-multus-daemon-config\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.163355 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163267 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-run\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.163355 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163284 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:53.163355 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163300 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-netns\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.163355 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163352 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-multus\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.163591 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163380 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-script-lib\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.163591 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163420 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/735421c1-678c-4cc4-a07e-57ffb0e9ca29-tmp-dir\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:53.163591 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163482 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-cni-binary-copy\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.163591 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163497 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/735421c1-678c-4cc4-a07e-57ffb0e9ca29-tmp-dir\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:53.163591 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163504 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j57h\" (UniqueName: \"kubernetes.io/projected/53101ce6-458d-4b8f-ac21-36679d929ba0-kube-api-access-9j57h\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.163591 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163525 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-conf\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.163591 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163542 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-script-lib\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.163591 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163550 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-slash\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.163931 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163596 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-socket-dir-parent\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.163931 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163643 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.163931 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163662 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79214604-184d-4eb5-b63e-d2a30209835c-cni-binary-copy\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.163931 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163694 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-modprobe-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.163931 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163736 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/527b2ac7-739e-4f53-b553-474cad1eb712-host\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:53.163931 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163774 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfwb\" (UniqueName: \"kubernetes.io/projected/89a526ea-68b3-40e3-a474-02fa8cec0ab0-kube-api-access-bdfwb\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.163931 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163797 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.163931 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163828 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-sys-fs\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.163931 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163928 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-cnibin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.163962 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/527b2ac7-739e-4f53-b553-474cad1eb712-serviceca\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164032 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-kubelet\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164062 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-etc-kubernetes\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164079 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164103 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-systemd\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164133 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-sys\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164147 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-var-lib-kubelet\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164181 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh7g\" (UniqueName: \"kubernetes.io/projected/92283fa1-fe9a-4287-a313-176acb992dc9-kube-api-access-dbh7g\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164195 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/527b2ac7-739e-4f53-b553-474cad1eb712-serviceca\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164204 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/53101ce6-458d-4b8f-ac21-36679d929ba0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164224 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164251 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a18dfddd-46cd-48c6-898c-0b313c829499-host-slash\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164270 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:53.164335 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164306 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-socket-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164358 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfc7z\" (UniqueName: \"kubernetes.io/projected/67af9038-04a3-4574-96fd-23a3bc52d944-kube-api-access-vfc7z\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164385 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-hostroot\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164478 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ee5c83e-ba60-41ef-a638-a77cdff3c824-original-pull-secret\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164480 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-kubernetes\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164508 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-etc-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164540 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164575 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-systemd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164604 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-node-log\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164619 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164638 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-bin\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164671 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-netd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164718 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/735421c1-678c-4cc4-a07e-57ffb0e9ca29-hosts-file\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164774 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-kubelet\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164806 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nccm2\" (UniqueName: \"kubernetes.io/projected/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-kube-api-access-nccm2\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164828 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-systemd-units\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.164874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164880 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-host\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164904 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtt6c\" (UniqueName: \"kubernetes.io/projected/a18dfddd-46cd-48c6-898c-0b313c829499-kube-api-access-dtt6c\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.164938 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-device-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165031 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165059 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqfz5\" (UniqueName: \"kubernetes.io/projected/79214604-184d-4eb5-b63e-d2a30209835c-kube-api-access-mqfz5\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165080 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-cnibin\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165110 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165128 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-ovn\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165143 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165170 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-registration-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165214 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-bin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165273 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-conf-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165309 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10158047-423e-4241-b46d-6ad29dece441-konnectivity-ca\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165344 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-tmp\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165367 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-log-socket\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165418 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-config\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165443 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92283fa1-fe9a-4287-a313-176acb992dc9-tmp\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165447 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-multus-certs\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165485 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10158047-423e-4241-b46d-6ad29dece441-konnectivity-ca\") pod \"konnectivity-agent-zxdmr\" (UID: \"10158047-423e-4241-b46d-6ad29dece441\") " pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:57:53.165531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165516 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysconfig\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.166144 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.165670 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/89a526ea-68b3-40e3-a474-02fa8cec0ab0-ovnkube-config\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.169712 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.169687 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsdx\" (UniqueName: \"kubernetes.io/projected/735421c1-678c-4cc4-a07e-57ffb0e9ca29-kube-api-access-wtsdx\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:53.170078 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.170061 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqjx\" (UniqueName: \"kubernetes.io/projected/527b2ac7-739e-4f53-b553-474cad1eb712-kube-api-access-6rqjx\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:53.170244 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.170229 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j57h\" (UniqueName: \"kubernetes.io/projected/53101ce6-458d-4b8f-ac21-36679d929ba0-kube-api-access-9j57h\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.170411 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.170361 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfwb\" (UniqueName: \"kubernetes.io/projected/89a526ea-68b3-40e3-a474-02fa8cec0ab0-kube-api-access-bdfwb\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.170442 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.170406 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjzw\" (UniqueName: \"kubernetes.io/projected/c5023155-869d-44af-9168-7a44f3cdc1a3-kube-api-access-pvjzw\") pod \"network-check-target-pft59\" (UID: \"c5023155-869d-44af-9168-7a44f3cdc1a3\") " pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:53.171113 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.171098 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbh7g\" (UniqueName: \"kubernetes.io/projected/92283fa1-fe9a-4287-a313-176acb992dc9-kube-api-access-dbh7g\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.171273 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.171254 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqfz5\" (UniqueName: \"kubernetes.io/projected/79214604-184d-4eb5-b63e-d2a30209835c-kube-api-access-mqfz5\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.171314 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.171254 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfc7z\" (UniqueName: \"kubernetes.io/projected/67af9038-04a3-4574-96fd-23a3bc52d944-kube-api-access-vfc7z\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.171574 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.171558 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtt6c\" (UniqueName: \"kubernetes.io/projected/a18dfddd-46cd-48c6-898c-0b313c829499-kube-api-access-dtt6c\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:53.171604 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.171569 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nccm2\" (UniqueName: \"kubernetes.io/projected/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-kube-api-access-nccm2\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:53.213019 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.212995 4309 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:53.213154 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.213101 4309 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:53.224509 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.224487 4309 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:57:53.224610 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:57:53.224524 4309 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:53.224671 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.224608 4309 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 21:57:53.224671 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:57:53.224640 4309 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-9.us-east-2.compute.internal" Nov 02 21:57:53.266570 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266545 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-netd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266578 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/735421c1-678c-4cc4-a07e-57ffb0e9ca29-hosts-file\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266602 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-kubelet\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266624 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-systemd-units\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266643 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-host\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266659 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-netd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266664 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-device-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266667 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/735421c1-678c-4cc4-a07e-57ffb0e9ca29-hosts-file\") pod \"node-resolver-kclzj\" (UID: \"735421c1-678c-4cc4-a07e-57ffb0e9ca29\") " pod="openshift-dns/node-resolver-kclzj" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266691 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-kubelet\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266721 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266729 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-host\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.266750 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266743 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-device-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266766 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266700 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-systemd-units\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266885 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-cnibin\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266909 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266924 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-ovn\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266931 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-cnibin\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266938 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266978 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.266978 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-registration-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267011 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267018 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-bin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267010 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-ovn\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267026 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-registration-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267047 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-conf-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267051 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-bin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267075 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-log-socket\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267093 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-multus-certs\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267126 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-log-socket\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267129 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-conf-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267127 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-multus-certs\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267141 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysconfig\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267175 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysconfig\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267177 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-k8s-cni-cncf-io\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267204 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.267203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267220 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-k8s-cni-cncf-io\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267226 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-lib-modules\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267248 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/53101ce6-458d-4b8f-ac21-36679d929ba0-system-cni-dir\") pod \"multus-additional-cni-plugins-6jc58\" (UID: \"53101ce6-458d-4b8f-ac21-36679d929ba0\") " pod="openshift-multus/multus-additional-cni-plugins-6jc58" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267249 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-system-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267273 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-netns\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267299 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-dbus\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267322 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-kubelet-config\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267328 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-lib-modules\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267340 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-system-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267346 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267346 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-netns\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267371 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-kubelet-config\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267372 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267383 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ee5c83e-ba60-41ef-a638-a77cdff3c824-dbus\") pod \"global-pull-secret-syncer-qkl72\" (UID: \"3ee5c83e-ba60-41ef-a638-a77cdff3c824\") " pod="kube-system/global-pull-secret-syncer-qkl72" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267401 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-var-lib-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267416 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-cni-dir\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267438 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-var-lib-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267446 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267438 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267466 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267476 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-run\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267494 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-netns\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267508 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-multus\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267544 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-run-netns\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267554 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-run\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267558 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-conf\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267587 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-slash\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267613 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-socket-dir-parent\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267624 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-host-var-lib-cni-multus\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267653 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-sysctl-conf\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267662 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-slash\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267662 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-modprobe-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267711 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/527b2ac7-739e-4f53-b553-474cad1eb712-host\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267730 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-multus-socket-dir-parent\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267748 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-sys-fs\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267758 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/527b2ac7-739e-4f53-b553-474cad1eb712-host\") pod \"node-ca-jg7pl\" (UID: \"527b2ac7-739e-4f53-b553-474cad1eb712\") " pod="openshift-image-registry/node-ca-jg7pl" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267772 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-cnibin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267775 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-modprobe-d\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267789 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-sys-fs\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267796 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-kubelet\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267820 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-cnibin\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267821 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-etc-kubernetes\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267845 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-systemd\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267845 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-kubelet\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267859 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-etc-kubernetes\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267861 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-sys\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267884 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-systemd\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267891 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-sys\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267897 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-var-lib-kubelet\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267923 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267948 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a18dfddd-46cd-48c6-898c-0b313c829499-host-slash\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267953 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-var-lib-kubelet\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267972 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-socket-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267983 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a18dfddd-46cd-48c6-898c-0b313c829499-host-slash\") pod \"iptables-alerter-lqhnn\" (UID: \"a18dfddd-46cd-48c6-898c-0b313c829499\") " pod="openshift-network-operator/iptables-alerter-lqhnn" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.267997 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-hostroot\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268019 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-kubernetes\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268042 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79214604-184d-4eb5-b63e-d2a30209835c-hostroot\") pod \"multus-dg6dc\" (UID: \"79214604-184d-4eb5-b63e-d2a30209835c\") " pod="openshift-multus/multus-dg6dc" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268043 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-etc-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268021 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/67af9038-04a3-4574-96fd-23a3bc52d944-socket-dir\") pod \"aws-ebs-csi-driver-node-nwzst\" (UID: \"67af9038-04a3-4574-96fd-23a3bc52d944\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwzst" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268066 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268084 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92283fa1-fe9a-4287-a313-176acb992dc9-etc-kubernetes\") pod \"tuned-rszqx\" (UID: \"92283fa1-fe9a-4287-a313-176acb992dc9\") " pod="openshift-cluster-node-tuning-operator/tuned-rszqx" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268090 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-systemd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268095 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-etc-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268119 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-node-log\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268121 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-openvswitch\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268135 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268135 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-run-systemd\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268165 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-node-log\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268166 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-run-ovn-kubernetes\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268164 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-bin\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268190 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/89a526ea-68b3-40e3-a474-02fa8cec0ab0-host-cni-bin\") pod \"ovnkube-node-pgrkb\" (UID: \"89a526ea-68b3-40e3-a474-02fa8cec0ab0\") " pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.268268 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.268271 4309 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 21:57:53.270258 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.270190 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a-metrics-certs\") pod \"network-metrics-daemon-4l48q\" (UID: \"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a\") " pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:53.448509 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.448429 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4l48q" Nov 02 21:57:53.599054 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.599027 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:53.721284 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.721211 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4l48q"] Nov 02 21:57:53.724425 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:53.724389 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5572bdbb_7cd4_4975_80d1_2fcc9c98fe5a.slice/crio-9f04cc0362dcd2e9404a527802e154edbd220376cc60b6540a62434f9e3ffa01 WatchSource:0}: Error finding container 9f04cc0362dcd2e9404a527802e154edbd220376cc60b6540a62434f9e3ffa01: Status 404 returned error can't find the container with id 9f04cc0362dcd2e9404a527802e154edbd220376cc60b6540a62434f9e3ffa01 Nov 02 21:57:53.726501 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:53.726483 4309 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 21:57:54.216324 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:54.216283 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4l48q" event={"ID":"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a","Type":"ContainerStarted","Data":"9f04cc0362dcd2e9404a527802e154edbd220376cc60b6540a62434f9e3ffa01"} Nov 02 21:57:56.222414 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:56.222380 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4l48q" event={"ID":"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a","Type":"ContainerStarted","Data":"6a95a999d95fcb15df2a872fb157fb190ed813694168dfdbfe2fc0a543a41585"} Nov 02 21:57:56.222799 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:56.222431 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4l48q" event={"ID":"5572bdbb-7cd4-4975-80d1-2fcc9c98fe5a","Type":"ContainerStarted","Data":"c0c9d1a585c56fdf147069745bba2e2efaa34ac0c5ff3a27d401db7e9f74c887"} Nov 02 21:57:56.680852 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:56.680824 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:57:56.681097 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:56.681012 4309 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:57:57.466969 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:57.466916 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:57:58.557118 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.557091 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qk42s"] Nov 02 21:57:58.558862 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.558846 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cgm5k"] Nov 02 21:57:58.558981 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.558963 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.560384 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.560367 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cgm5k" Nov 02 21:57:58.565079 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.565039 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-z8gn7"] Nov 02 21:57:58.566591 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.566577 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5"] Nov 02 21:57:58.566727 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.566711 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:57:58.567102 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.567084 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 21:57:58.568220 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.568206 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" Nov 02 21:57:58.573695 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.573660 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-b8hrl\"" Nov 02 21:57:58.573792 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.573707 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 21:57:58.573792 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.573780 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqhdf\"" Nov 02 21:57:58.574360 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.574343 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-zkn74\"" Nov 02 21:57:58.574456 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.574377 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qk42s"] Nov 02 21:57:58.574596 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.574578 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cgm5k"] Nov 02 21:57:58.577427 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.577405 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 21:57:58.577582 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.577564 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 21:57:58.577669 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.577612 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 21:57:58.577744 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.577700 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 21:57:58.578815 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.578793 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 21:57:58.583251 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.583227 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-z8gn7"] Nov 02 21:57:58.587196 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.587169 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5"] Nov 02 21:57:58.594339 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.594321 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsq7\" (UniqueName: \"kubernetes.io/projected/660abcf1-bcc1-486a-a0f1-dec089f5cc42-kube-api-access-vrsq7\") pod \"downloads-6dd789c8b7-z8gn7\" (UID: \"660abcf1-bcc1-486a-a0f1-dec089f5cc42\") " pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:57:58.594433 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.594346 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6894f559-b442-486b-a81d-05e13031bf22-metrics-tls\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.594433 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.594365 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/256801e0-d5c5-4928-8aed-8b86856fc560-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-26cj5\" (UID: \"256801e0-d5c5-4928-8aed-8b86856fc560\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" Nov 02 21:57:58.594433 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.594390 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/660abcf1-bcc1-486a-a0f1-dec089f5cc42-tmp\") pod \"downloads-6dd789c8b7-z8gn7\" (UID: \"660abcf1-bcc1-486a-a0f1-dec089f5cc42\") " pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:57:58.594545 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.594435 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6894f559-b442-486b-a81d-05e13031bf22-config-volume\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.594586 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.594561 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2a56b64-0440-4f3a-abc8-fa257a8db420-cert\") pod \"ingress-canary-cgm5k\" (UID: \"d2a56b64-0440-4f3a-abc8-fa257a8db420\") " pod="openshift-ingress-canary/ingress-canary-cgm5k" Nov 02 21:57:58.594622 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.594588 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9hw\" (UniqueName: \"kubernetes.io/projected/6894f559-b442-486b-a81d-05e13031bf22-kube-api-access-qs9hw\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.594660 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.594627 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjdt\" (UniqueName: \"kubernetes.io/projected/d2a56b64-0440-4f3a-abc8-fa257a8db420-kube-api-access-jrjdt\") pod \"ingress-canary-cgm5k\" (UID: \"d2a56b64-0440-4f3a-abc8-fa257a8db420\") " pod="openshift-ingress-canary/ingress-canary-cgm5k" Nov 02 21:57:58.594660 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.594643 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6894f559-b442-486b-a81d-05e13031bf22-tmp-dir\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.597045 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.597027 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hjd95"] Nov 02 21:57:58.598638 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.598625 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.605726 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.605704 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 21:57:58.605916 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.605902 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 21:57:58.605976 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.605963 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 21:57:58.610620 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.610602 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjd95"] Nov 02 21:57:58.670882 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.670859 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 21:57:58.670992 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.670899 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 21:57:58.670992 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.670984 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jcgv2\"" Nov 02 21:57:58.671068 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.670996 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9v8fd\"" Nov 02 21:57:58.695712 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.695687 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjdt\" (UniqueName: \"kubernetes.io/projected/d2a56b64-0440-4f3a-abc8-fa257a8db420-kube-api-access-jrjdt\") pod \"ingress-canary-cgm5k\" (UID: \"d2a56b64-0440-4f3a-abc8-fa257a8db420\") " pod="openshift-ingress-canary/ingress-canary-cgm5k" Nov 02 21:57:58.695801 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.695728 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6894f559-b442-486b-a81d-05e13031bf22-tmp-dir\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.695801 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.695760 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c69c738c-eb34-47f9-99b6-5770eab13da4-data-volume\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.695801 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.695790 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsq7\" (UniqueName: \"kubernetes.io/projected/660abcf1-bcc1-486a-a0f1-dec089f5cc42-kube-api-access-vrsq7\") pod \"downloads-6dd789c8b7-z8gn7\" (UID: \"660abcf1-bcc1-486a-a0f1-dec089f5cc42\") " pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:57:58.696002 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.695984 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6894f559-b442-486b-a81d-05e13031bf22-metrics-tls\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.696033 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696019 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/256801e0-d5c5-4928-8aed-8b86856fc560-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-26cj5\" (UID: \"256801e0-d5c5-4928-8aed-8b86856fc560\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" Nov 02 21:57:58.696062 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696048 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c69c738c-eb34-47f9-99b6-5770eab13da4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.696096 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696066 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c69c738c-eb34-47f9-99b6-5770eab13da4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.696161 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696145 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6894f559-b442-486b-a81d-05e13031bf22-tmp-dir\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.696195 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696142 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c69c738c-eb34-47f9-99b6-5770eab13da4-crio-socket\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.696227 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696206 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/660abcf1-bcc1-486a-a0f1-dec089f5cc42-tmp\") pod \"downloads-6dd789c8b7-z8gn7\" (UID: \"660abcf1-bcc1-486a-a0f1-dec089f5cc42\") " pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:57:58.696258 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696235 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6894f559-b442-486b-a81d-05e13031bf22-config-volume\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.696301 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696267 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2a56b64-0440-4f3a-abc8-fa257a8db420-cert\") pod \"ingress-canary-cgm5k\" (UID: \"d2a56b64-0440-4f3a-abc8-fa257a8db420\") " pod="openshift-ingress-canary/ingress-canary-cgm5k" Nov 02 21:57:58.696301 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696293 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9hw\" (UniqueName: \"kubernetes.io/projected/6894f559-b442-486b-a81d-05e13031bf22-kube-api-access-qs9hw\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.696377 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696326 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpl84\" (UniqueName: \"kubernetes.io/projected/c69c738c-eb34-47f9-99b6-5770eab13da4-kube-api-access-bpl84\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.696510 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696489 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/660abcf1-bcc1-486a-a0f1-dec089f5cc42-tmp\") pod \"downloads-6dd789c8b7-z8gn7\" (UID: \"660abcf1-bcc1-486a-a0f1-dec089f5cc42\") " pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:57:58.696724 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.696708 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6894f559-b442-486b-a81d-05e13031bf22-config-volume\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.698408 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.698383 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6894f559-b442-486b-a81d-05e13031bf22-metrics-tls\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.698528 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.698498 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/256801e0-d5c5-4928-8aed-8b86856fc560-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-26cj5\" (UID: \"256801e0-d5c5-4928-8aed-8b86856fc560\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" Nov 02 21:57:58.698583 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.698570 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2a56b64-0440-4f3a-abc8-fa257a8db420-cert\") pod \"ingress-canary-cgm5k\" (UID: \"d2a56b64-0440-4f3a-abc8-fa257a8db420\") " pod="openshift-ingress-canary/ingress-canary-cgm5k" Nov 02 21:57:58.708902 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.708876 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9hw\" (UniqueName: \"kubernetes.io/projected/6894f559-b442-486b-a81d-05e13031bf22-kube-api-access-qs9hw\") pod \"dns-default-qk42s\" (UID: \"6894f559-b442-486b-a81d-05e13031bf22\") " pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.709141 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.709125 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsq7\" (UniqueName: \"kubernetes.io/projected/660abcf1-bcc1-486a-a0f1-dec089f5cc42-kube-api-access-vrsq7\") pod \"downloads-6dd789c8b7-z8gn7\" (UID: \"660abcf1-bcc1-486a-a0f1-dec089f5cc42\") " pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:57:58.709304 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.709283 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjdt\" (UniqueName: \"kubernetes.io/projected/d2a56b64-0440-4f3a-abc8-fa257a8db420-kube-api-access-jrjdt\") pod \"ingress-canary-cgm5k\" (UID: \"d2a56b64-0440-4f3a-abc8-fa257a8db420\") " pod="openshift-ingress-canary/ingress-canary-cgm5k" Nov 02 21:57:58.796804 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.796779 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpl84\" (UniqueName: \"kubernetes.io/projected/c69c738c-eb34-47f9-99b6-5770eab13da4-kube-api-access-bpl84\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.796930 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.796810 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c69c738c-eb34-47f9-99b6-5770eab13da4-data-volume\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.796930 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.796838 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c69c738c-eb34-47f9-99b6-5770eab13da4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.796930 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.796857 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c69c738c-eb34-47f9-99b6-5770eab13da4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.796930 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.796882 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c69c738c-eb34-47f9-99b6-5770eab13da4-crio-socket\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.797161 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.796957 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c69c738c-eb34-47f9-99b6-5770eab13da4-crio-socket\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.797216 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.797191 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c69c738c-eb34-47f9-99b6-5770eab13da4-data-volume\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.797368 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.797348 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c69c738c-eb34-47f9-99b6-5770eab13da4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.798900 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.798882 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c69c738c-eb34-47f9-99b6-5770eab13da4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.804759 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.804740 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpl84\" (UniqueName: \"kubernetes.io/projected/c69c738c-eb34-47f9-99b6-5770eab13da4-kube-api-access-bpl84\") pod \"insights-runtime-extractor-hjd95\" (UID: \"c69c738c-eb34-47f9-99b6-5770eab13da4\") " pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:58.870656 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.870632 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qk42s" Nov 02 21:57:58.877981 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.877961 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cgm5k" Nov 02 21:57:58.885716 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.885696 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:57:58.890535 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.890512 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" Nov 02 21:57:58.913596 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:58.913559 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjd95" Nov 02 21:57:59.141653 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:59.141590 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qk42s"] Nov 02 21:57:59.144209 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:59.144188 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6894f559_b442_486b_a81d_05e13031bf22.slice/crio-4cd632de3089faa3c16b15251501fe3935eea8058728e72cc61cdbe34d083a97 WatchSource:0}: Error finding container 4cd632de3089faa3c16b15251501fe3935eea8058728e72cc61cdbe34d083a97: Status 404 returned error can't find the container with id 4cd632de3089faa3c16b15251501fe3935eea8058728e72cc61cdbe34d083a97 Nov 02 21:57:59.166756 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:59.166733 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cgm5k"] Nov 02 21:57:59.169195 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:59.169173 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a56b64_0440_4f3a_abc8_fa257a8db420.slice/crio-009e903d3dc6b5901327f54465af12f6bb1e4dbe7fe2d9877157695cbdb85ef9 WatchSource:0}: Error finding container 009e903d3dc6b5901327f54465af12f6bb1e4dbe7fe2d9877157695cbdb85ef9: Status 404 returned error can't find the container with id 009e903d3dc6b5901327f54465af12f6bb1e4dbe7fe2d9877157695cbdb85ef9 Nov 02 21:57:59.197889 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:59.197867 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-z8gn7"] Nov 02 21:57:59.200512 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:59.200488 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5"] Nov 02 21:57:59.200603 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:59.200552 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660abcf1_bcc1_486a_a0f1_dec089f5cc42.slice/crio-c6bc2a919a47d5db379d13d9ade6f2adde4a810e722c4f9128b9eb4bb7c14010 WatchSource:0}: Error finding container c6bc2a919a47d5db379d13d9ade6f2adde4a810e722c4f9128b9eb4bb7c14010: Status 404 returned error can't find the container with id c6bc2a919a47d5db379d13d9ade6f2adde4a810e722c4f9128b9eb4bb7c14010 Nov 02 21:57:59.204773 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:59.204656 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256801e0_d5c5_4928_8aed_8b86856fc560.slice/crio-cf33103085013ba84462592e343d8d5ba2656c0c627cb3dc28784a6c30c73126 WatchSource:0}: Error finding container cf33103085013ba84462592e343d8d5ba2656c0c627cb3dc28784a6c30c73126: Status 404 returned error can't find the container with id cf33103085013ba84462592e343d8d5ba2656c0c627cb3dc28784a6c30c73126 Nov 02 21:57:59.229261 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:59.229238 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cgm5k" event={"ID":"d2a56b64-0440-4f3a-abc8-fa257a8db420","Type":"ContainerStarted","Data":"009e903d3dc6b5901327f54465af12f6bb1e4dbe7fe2d9877157695cbdb85ef9"} Nov 02 21:57:59.230284 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:59.230264 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qk42s" event={"ID":"6894f559-b442-486b-a81d-05e13031bf22","Type":"ContainerStarted","Data":"4cd632de3089faa3c16b15251501fe3935eea8058728e72cc61cdbe34d083a97"} Nov 02 21:57:59.231342 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:59.231323 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" event={"ID":"256801e0-d5c5-4928-8aed-8b86856fc560","Type":"ContainerStarted","Data":"cf33103085013ba84462592e343d8d5ba2656c0c627cb3dc28784a6c30c73126"} Nov 02 21:57:59.232351 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:59.232333 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-z8gn7" event={"ID":"660abcf1-bcc1-486a-a0f1-dec089f5cc42","Type":"ContainerStarted","Data":"c6bc2a919a47d5db379d13d9ade6f2adde4a810e722c4f9128b9eb4bb7c14010"} Nov 02 21:57:59.262124 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:57:59.262097 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjd95"] Nov 02 21:57:59.265165 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:57:59.265147 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69c738c_eb34_47f9_99b6_5770eab13da4.slice/crio-8883bc3ef3211ec59afad2cbcf48a5152a4f422f51b48e48aefe32247df17f15 WatchSource:0}: Error finding container 8883bc3ef3211ec59afad2cbcf48a5152a4f422f51b48e48aefe32247df17f15: Status 404 returned error can't find the container with id 8883bc3ef3211ec59afad2cbcf48a5152a4f422f51b48e48aefe32247df17f15 Nov 02 21:58:00.237372 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:00.237331 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjd95" event={"ID":"c69c738c-eb34-47f9-99b6-5770eab13da4","Type":"ContainerStarted","Data":"a2e9c83b001d1e78c283aa8e0c00c2fdd94ffad7d075f18a03b9359ef24f1263"} Nov 02 21:58:00.237762 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:00.237401 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjd95" event={"ID":"c69c738c-eb34-47f9-99b6-5770eab13da4","Type":"ContainerStarted","Data":"7365d5a31e13ad7ba19618a36d4bdb6f279a93a7e8185312a866120d495840db"} Nov 02 21:58:00.237762 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:00.237426 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjd95" event={"ID":"c69c738c-eb34-47f9-99b6-5770eab13da4","Type":"ContainerStarted","Data":"8883bc3ef3211ec59afad2cbcf48a5152a4f422f51b48e48aefe32247df17f15"} Nov 02 21:58:02.246599 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:02.246513 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" event={"ID":"256801e0-d5c5-4928-8aed-8b86856fc560","Type":"ContainerStarted","Data":"9a958bae733343b3c59ed058535a9f94f458ae666d6305eae6048d617ab4ea65"} Nov 02 21:58:02.249926 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:02.249871 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjd95" event={"ID":"c69c738c-eb34-47f9-99b6-5770eab13da4","Type":"ContainerStarted","Data":"66fac139b960338e75899f85a85ba3f9bca2c34f79be73f20ef37b964f5ccf52"} Nov 02 21:58:02.262149 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:02.262085 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" podStartSLOduration=2.181183497 podStartE2EDuration="4.262068944s" totalImagesPullingTime="2.080885447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:02.261845437 +0000 UTC m=+10.199958657" watchObservedRunningTime="2026-11-02 21:58:02.262068944 +0000 UTC m=+10.200182162" Nov 02 21:58:02.284048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:02.283985 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hjd95" podStartSLOduration=1.7516837669999998 podStartE2EDuration="4.28396169s" totalImagesPullingTime="2.532277923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:57:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:02.279086723 +0000 UTC m=+10.217199941" watchObservedRunningTime="2026-11-02 21:58:02.28396169 +0000 UTC m=+10.222074963" Nov 02 21:58:03.255637 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:03.255591 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cgm5k" event={"ID":"d2a56b64-0440-4f3a-abc8-fa257a8db420","Type":"ContainerStarted","Data":"49e482d2e98fa4172c16988e54aa9dacacea16c0d75a0a261c7121fd47650264"} Nov 02 21:58:03.256142 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:03.256118 4309 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:58:03.272504 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:03.272458 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cgm5k" podStartSLOduration=1.6975991129999999 podStartE2EDuration="5.272445543s" totalImagesPullingTime="3.57484643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:03.272064151 +0000 UTC m=+11.210177368" watchObservedRunningTime="2026-11-02 21:58:03.272445543 +0000 UTC m=+11.210558761" Nov 02 21:58:06.050847 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.050807 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" Nov 02 21:58:06.051294 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.050936 4309 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 21:58:06.056493 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.056469 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-26cj5" Nov 02 21:58:06.268289 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.268251 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qk42s" event={"ID":"6894f559-b442-486b-a81d-05e13031bf22","Type":"ContainerStarted","Data":"fd3b480b297ab84d7f1e20581a6a86f78afc1dbd6db94d79a5a6c42caf1db33a"} Nov 02 21:58:06.268468 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.268321 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qk42s" event={"ID":"6894f559-b442-486b-a81d-05e13031bf22","Type":"ContainerStarted","Data":"756909185a32fdd1fa51a28c305f0e2baab24b230ed282cf2cf8beeb739acd7c"} Nov 02 21:58:06.268602 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.268586 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qk42s" Nov 02 21:58:06.285169 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.285118 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-qk42s" podStartSLOduration=2.244696383 podStartE2EDuration="8.285101297s" totalImagesPullingTime="6.040404914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:06.283840569 +0000 UTC m=+14.221953818" watchObservedRunningTime="2026-11-02 21:58:06.285101297 +0000 UTC m=+14.223214512" Nov 02 21:58:06.463759 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.462539 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj"] Nov 02 21:58:06.465988 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.465966 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.474362 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.474324 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 21:58:06.475065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.475040 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj"] Nov 02 21:58:06.475469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.475451 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 21:58:06.475577 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.475469 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 21:58:06.475577 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.475478 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 21:58:06.475667 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.475598 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 21:58:06.475667 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.475647 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-f5vqn\"" Nov 02 21:58:06.555425 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.555390 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32361f80-c746-4c97-9e73-20798d0c5417-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.555586 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.555446 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32361f80-c746-4c97-9e73-20798d0c5417-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.555586 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.555546 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tksb5\" (UniqueName: \"kubernetes.io/projected/32361f80-c746-4c97-9e73-20798d0c5417-kube-api-access-tksb5\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.555729 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.555613 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32361f80-c746-4c97-9e73-20798d0c5417-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.656065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.656029 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32361f80-c746-4c97-9e73-20798d0c5417-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.656065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.656067 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tksb5\" (UniqueName: \"kubernetes.io/projected/32361f80-c746-4c97-9e73-20798d0c5417-kube-api-access-tksb5\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.656270 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.656122 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32361f80-c746-4c97-9e73-20798d0c5417-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.656270 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.656153 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32361f80-c746-4c97-9e73-20798d0c5417-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.656911 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.656883 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32361f80-c746-4c97-9e73-20798d0c5417-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.659306 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.659280 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/32361f80-c746-4c97-9e73-20798d0c5417-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.662119 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.662092 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32361f80-c746-4c97-9e73-20798d0c5417-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.664114 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.664083 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tksb5\" (UniqueName: \"kubernetes.io/projected/32361f80-c746-4c97-9e73-20798d0c5417-kube-api-access-tksb5\") pod \"prometheus-operator-9cd89b74c-rt7kj\" (UID: \"32361f80-c746-4c97-9e73-20798d0c5417\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:06.781920 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:06.781841 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" Nov 02 21:58:07.058446 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:07.058417 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj"] Nov 02 21:58:07.061959 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:58:07.061930 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32361f80_c746_4c97_9e73_20798d0c5417.slice/crio-8215174ba63dbc5af1498239ad5cf3d7b4ef80bf8d07cf1b088a1c0f548eb888 WatchSource:0}: Error finding container 8215174ba63dbc5af1498239ad5cf3d7b4ef80bf8d07cf1b088a1c0f548eb888: Status 404 returned error can't find the container with id 8215174ba63dbc5af1498239ad5cf3d7b4ef80bf8d07cf1b088a1c0f548eb888 Nov 02 21:58:07.273231 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:07.273195 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" event={"ID":"32361f80-c746-4c97-9e73-20798d0c5417","Type":"ContainerStarted","Data":"8215174ba63dbc5af1498239ad5cf3d7b4ef80bf8d07cf1b088a1c0f548eb888"} Nov 02 21:58:08.398048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:08.398013 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zxdmr" Nov 02 21:58:10.285162 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:10.285129 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" event={"ID":"32361f80-c746-4c97-9e73-20798d0c5417","Type":"ContainerStarted","Data":"9bcd9092b816aa2fee924d4e9538be822643772ee8e8df737c481b8c132f1e89"} Nov 02 21:58:10.285590 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:10.285224 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" event={"ID":"32361f80-c746-4c97-9e73-20798d0c5417","Type":"ContainerStarted","Data":"e08af5d37692e0135ed810ebb15659a639115333c6b13ce761135ea29fdb69b8"} Nov 02 21:58:10.301972 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:10.301914 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rt7kj" podStartSLOduration=1.800441868 podStartE2EDuration="4.30189719s" totalImagesPullingTime="2.501455322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:10.300581769 +0000 UTC m=+18.238694986" watchObservedRunningTime="2026-11-02 21:58:10.30189719 +0000 UTC m=+18.240010411" Nov 02 21:58:11.598226 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:58:11.598194 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 21:58:11.817464 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.816365 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv"] Nov 02 21:58:11.819871 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.819846 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.827217 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.827195 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 21:58:11.827480 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.827461 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 21:58:11.829626 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.829602 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 21:58:11.829726 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.829713 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v976j\"" Nov 02 21:58:11.830109 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.830089 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bzm9b"] Nov 02 21:58:11.833315 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.833296 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv"] Nov 02 21:58:11.833448 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.833427 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.867208 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.867153 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xgmtc\"" Nov 02 21:58:11.867208 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.867173 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 21:58:11.867360 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.867238 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 21:58:11.893622 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.893591 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a657025-9495-432b-a966-db8b41cae3df-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.893737 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.893637 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b08273d-5f57-44e7-896d-61ee487fad84-sys\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.893737 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.893666 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq98q\" (UniqueName: \"kubernetes.io/projected/9b08273d-5f57-44e7-896d-61ee487fad84-kube-api-access-tq98q\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.893842 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.893767 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-accelerators-collector-config\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.893842 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.893809 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0a657025-9495-432b-a966-db8b41cae3df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.893936 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.893854 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a657025-9495-432b-a966-db8b41cae3df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.893936 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.893922 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b08273d-5f57-44e7-896d-61ee487fad84-root\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.894018 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.893966 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.894018 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.893999 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a657025-9495-432b-a966-db8b41cae3df-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.894107 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.894031 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b08273d-5f57-44e7-896d-61ee487fad84-metrics-client-ca\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.894146 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.894101 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0a657025-9495-432b-a966-db8b41cae3df-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.894192 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.894154 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-textfile\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.894231 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.894213 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642gx\" (UniqueName: \"kubernetes.io/projected/0a657025-9495-432b-a966-db8b41cae3df-kube-api-access-642gx\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.894272 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.894237 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-tls\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.894321 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.894299 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-wtmp\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.927712 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.927670 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 21:58:11.995229 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995192 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b08273d-5f57-44e7-896d-61ee487fad84-metrics-client-ca\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995229 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995232 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0a657025-9495-432b-a966-db8b41cae3df-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.995438 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995265 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-textfile\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995499 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995461 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642gx\" (UniqueName: \"kubernetes.io/projected/0a657025-9495-432b-a966-db8b41cae3df-kube-api-access-642gx\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.995555 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995510 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-tls\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995555 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995539 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-wtmp\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995663 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995569 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a657025-9495-432b-a966-db8b41cae3df-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.995663 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995600 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b08273d-5f57-44e7-896d-61ee487fad84-sys\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995663 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995610 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-textfile\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995663 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995627 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq98q\" (UniqueName: \"kubernetes.io/projected/9b08273d-5f57-44e7-896d-61ee487fad84-kube-api-access-tq98q\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995666 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-accelerators-collector-config\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995692 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0a657025-9495-432b-a966-db8b41cae3df-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.995880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995723 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0a657025-9495-432b-a966-db8b41cae3df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.995880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995757 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-wtmp\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995770 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a657025-9495-432b-a966-db8b41cae3df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.995880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995798 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b08273d-5f57-44e7-896d-61ee487fad84-root\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995825 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.995880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995852 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a657025-9495-432b-a966-db8b41cae3df-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.995880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995868 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b08273d-5f57-44e7-896d-61ee487fad84-metrics-client-ca\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.996350 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.995926 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b08273d-5f57-44e7-896d-61ee487fad84-sys\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.996350 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.996170 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9b08273d-5f57-44e7-896d-61ee487fad84-root\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.997350 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.996728 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a657025-9495-432b-a966-db8b41cae3df-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.997350 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.996747 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-accelerators-collector-config\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.997350 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.997311 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0a657025-9495-432b-a966-db8b41cae3df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.998559 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.998539 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a657025-9495-432b-a966-db8b41cae3df-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.998638 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.998584 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-tls\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:11.998788 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.998761 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a657025-9495-432b-a966-db8b41cae3df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:11.998898 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:11.998864 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b08273d-5f57-44e7-896d-61ee487fad84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:12.003395 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.003371 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-642gx\" (UniqueName: \"kubernetes.io/projected/0a657025-9495-432b-a966-db8b41cae3df-kube-api-access-642gx\") pod \"kube-state-metrics-5dc8d7d89b-k6wjv\" (UID: \"0a657025-9495-432b-a966-db8b41cae3df\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:12.003876 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.003854 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq98q\" (UniqueName: \"kubernetes.io/projected/9b08273d-5f57-44e7-896d-61ee487fad84-kube-api-access-tq98q\") pod \"node-exporter-bzm9b\" (UID: \"9b08273d-5f57-44e7-896d-61ee487fad84\") " pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:12.139066 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.138989 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" Nov 02 21:58:12.150959 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.150936 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bzm9b" Nov 02 21:58:12.167922 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:58:12.167896 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b08273d_5f57_44e7_896d_61ee487fad84.slice/crio-ccb10fe8b0be586745050a6ec048b49fda63ea959508b26ac85f7a54489183eb WatchSource:0}: Error finding container ccb10fe8b0be586745050a6ec048b49fda63ea959508b26ac85f7a54489183eb: Status 404 returned error can't find the container with id ccb10fe8b0be586745050a6ec048b49fda63ea959508b26ac85f7a54489183eb Nov 02 21:58:12.292766 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.292726 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bzm9b" event={"ID":"9b08273d-5f57-44e7-896d-61ee487fad84","Type":"ContainerStarted","Data":"ccb10fe8b0be586745050a6ec048b49fda63ea959508b26ac85f7a54489183eb"} Nov 02 21:58:12.425641 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.425535 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv"] Nov 02 21:58:12.429106 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:58:12.429067 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a657025_9495_432b_a966_db8b41cae3df.slice/crio-19d84d7c4f69feacd1742798b31942b84e4b5071ba0608b41df86214d927e659 WatchSource:0}: Error finding container 19d84d7c4f69feacd1742798b31942b84e4b5071ba0608b41df86214d927e659: Status 404 returned error can't find the container with id 19d84d7c4f69feacd1742798b31942b84e4b5071ba0608b41df86214d927e659 Nov 02 21:58:12.876743 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.876708 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:58:12.880334 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.880308 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:12.884618 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.884596 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 21:58:12.884763 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.884648 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 21:58:12.884832 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.884805 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 21:58:12.884967 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.884948 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 21:58:12.885076 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.884994 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 21:58:12.885076 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.884996 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 21:58:12.885076 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.885026 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 21:58:12.885076 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.885050 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 21:58:12.885076 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.885062 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 21:58:12.885508 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.885489 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n2nmp\"" Nov 02 21:58:12.896416 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:12.896383 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:58:13.003174 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003139 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003317 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003197 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjlp\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-kube-api-access-2sjlp\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003317 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003221 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003317 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003245 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003317 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003261 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003317 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003297 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003319 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003347 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-out\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003390 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003416 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-web-config\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003438 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003470 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.003546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.003493 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.104602 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104562 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.104602 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104613 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjlp\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-kube-api-access-2sjlp\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.104874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104646 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.104874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104696 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.104874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104724 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.104874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104758 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.104874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104790 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.104874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104820 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-out\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.104874 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104858 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.105223 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104904 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-web-config\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.105223 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104935 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.105223 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.104968 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.105223 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.105001 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.105223 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.105068 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.105498 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:58:13.105477 4309 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle podName:f2879ff3-1cff-4772-8d5c-1c2818edcfef nodeName:}" failed. No retries permitted until 2026-11-02 21:58:13.605447827 +0000 UTC m=+21.543561045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef") : configmap references non-existent config key: ca-bundle.crt Nov 02 21:58:13.105629 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.105512 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.108307 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.108262 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.108532 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.108510 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-web-config\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.110353 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.110329 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-out\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.110454 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.110435 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.110529 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.110500 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.110588 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.110563 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.110784 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.110745 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-volume\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.112010 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.111987 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.116392 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.116373 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjlp\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-kube-api-access-2sjlp\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.116476 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.116419 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.297638 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.297556 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" event={"ID":"0a657025-9495-432b-a966-db8b41cae3df","Type":"ContainerStarted","Data":"19d84d7c4f69feacd1742798b31942b84e4b5071ba0608b41df86214d927e659"} Nov 02 21:58:13.323966 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.323928 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f7cbb8bcd-hccnd"] Nov 02 21:58:13.327177 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.327153 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.340031 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.340008 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 21:58:13.340228 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.340210 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 21:58:13.340445 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.340404 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 21:58:13.340575 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.340561 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 21:58:13.340706 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.340688 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 21:58:13.341837 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.341814 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j7lvq\"" Nov 02 21:58:13.341956 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.341929 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f7cbb8bcd-hccnd"] Nov 02 21:58:13.407552 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.407517 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-oauth-config\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.407759 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.407589 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-console-config\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.407759 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.407653 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmps\" (UniqueName: \"kubernetes.io/projected/391f0200-66e0-46a2-90c5-e34878a8b4af-kube-api-access-cmmps\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.407897 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.407765 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-service-ca\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.407897 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.407819 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-serving-cert\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.407897 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.407851 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-oauth-serving-cert\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.407897 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.407886 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/391f0200-66e0-46a2-90c5-e34878a8b4af-tmp\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.509087 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.509046 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-console-config\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.509281 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.509095 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmps\" (UniqueName: \"kubernetes.io/projected/391f0200-66e0-46a2-90c5-e34878a8b4af-kube-api-access-cmmps\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.509281 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.509143 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-service-ca\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.509281 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.509188 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-serving-cert\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.509281 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.509215 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-oauth-serving-cert\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.509281 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.509246 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/391f0200-66e0-46a2-90c5-e34878a8b4af-tmp\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.509281 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.509273 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-oauth-config\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.510016 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.509985 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-console-config\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.510225 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.510205 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-service-ca\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.510305 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.510288 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/391f0200-66e0-46a2-90c5-e34878a8b4af-tmp\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.510490 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.510467 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-oauth-serving-cert\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.512345 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.512321 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-serving-cert\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.515501 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.515473 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-oauth-config\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.517307 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.517269 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmps\" (UniqueName: \"kubernetes.io/projected/391f0200-66e0-46a2-90c5-e34878a8b4af-kube-api-access-cmmps\") pod \"console-f7cbb8bcd-hccnd\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.610219 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.610183 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.611023 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.611000 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.643013 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.642992 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:13.810603 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.810534 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:58:13.812349 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.812319 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75fcc597f8-zw9z8"] Nov 02 21:58:13.816651 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.816022 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.830122 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.830100 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75fcc597f8-zw9z8"] Nov 02 21:58:13.909825 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.909417 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 21:58:13.909825 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.909523 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 21:58:13.909825 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.909619 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 21:58:13.909825 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.909705 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3040onn32t42t\"" Nov 02 21:58:13.909825 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.909792 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wwbwl\"" Nov 02 21:58:13.909825 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.909808 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 21:58:13.909825 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.909818 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 21:58:13.915227 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.915136 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.915227 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.915194 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.915476 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.915284 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.915476 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.915311 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hkg\" (UniqueName: \"kubernetes.io/projected/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-kube-api-access-h4hkg\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.915476 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.915384 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-tls\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.915476 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.915423 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-tmp\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.915476 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.915472 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.915840 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.915529 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-metrics-client-ca\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.915840 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.915563 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-grpc-tls\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:13.932567 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:13.932537 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f7cbb8bcd-hccnd"] Nov 02 21:58:13.936305 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:58:13.936280 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391f0200_66e0_46a2_90c5_e34878a8b4af.slice/crio-3eeb1101b14cff587f026b0bc2eeb898afba3e033d39be01a81665b2f29464c5 WatchSource:0}: Error finding container 3eeb1101b14cff587f026b0bc2eeb898afba3e033d39be01a81665b2f29464c5: Status 404 returned error can't find the container with id 3eeb1101b14cff587f026b0bc2eeb898afba3e033d39be01a81665b2f29464c5 Nov 02 21:58:14.016190 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.016158 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-grpc-tls\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.016349 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.016202 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.016349 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.016313 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.016475 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.016358 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.016475 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.016385 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hkg\" (UniqueName: \"kubernetes.io/projected/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-kube-api-access-h4hkg\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.016475 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.016409 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-tls\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.016475 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.016440 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-tmp\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.016663 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.016494 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.016663 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.016547 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-metrics-client-ca\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.017357 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.017330 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-metrics-client-ca\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.017478 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.017376 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-tmp\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.019983 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.019957 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.020085 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.019992 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-grpc-tls\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.020626 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.020602 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.021014 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.020966 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.021316 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.021288 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.022106 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.022086 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-secret-thanos-querier-tls\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.025636 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.025617 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hkg\" (UniqueName: \"kubernetes.io/projected/c9ef1d8c-8213-455c-a988-925d8cdaa2b5-kube-api-access-h4hkg\") pod \"thanos-querier-75fcc597f8-zw9z8\" (UID: \"c9ef1d8c-8213-455c-a988-925d8cdaa2b5\") " pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.104119 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.104084 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:58:14.111041 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:58:14.111011 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2879ff3_1cff_4772_8d5c_1c2818edcfef.slice/crio-074b2e5852bd9b95deee6ad6138d6e9eb9e37886eb4ffb1aab18989408a2a812 WatchSource:0}: Error finding container 074b2e5852bd9b95deee6ad6138d6e9eb9e37886eb4ffb1aab18989408a2a812: Status 404 returned error can't find the container with id 074b2e5852bd9b95deee6ad6138d6e9eb9e37886eb4ffb1aab18989408a2a812 Nov 02 21:58:14.145143 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.145067 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:14.302622 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.302583 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7cbb8bcd-hccnd" event={"ID":"391f0200-66e0-46a2-90c5-e34878a8b4af","Type":"ContainerStarted","Data":"3eeb1101b14cff587f026b0bc2eeb898afba3e033d39be01a81665b2f29464c5"} Nov 02 21:58:14.303904 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.303873 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerStarted","Data":"074b2e5852bd9b95deee6ad6138d6e9eb9e37886eb4ffb1aab18989408a2a812"} Nov 02 21:58:14.305539 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.305518 4309 generic.go:356] "Generic (PLEG): container finished" podID="9b08273d-5f57-44e7-896d-61ee487fad84" containerID="d0f53e9bf7a154289ee27d0b616e785916f43f31c66c1e0b5f26ffe7eafb6aa7" exitCode=0 Nov 02 21:58:14.305645 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.305570 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bzm9b" event={"ID":"9b08273d-5f57-44e7-896d-61ee487fad84","Type":"ContainerDied","Data":"d0f53e9bf7a154289ee27d0b616e785916f43f31c66c1e0b5f26ffe7eafb6aa7"} Nov 02 21:58:14.435367 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:14.435341 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75fcc597f8-zw9z8"] Nov 02 21:58:14.442933 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:58:14.442895 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ef1d8c_8213_455c_a988_925d8cdaa2b5.slice/crio-03713469979a0430fe1577b7d9a43e1136c3b092e19ca2cdf8d1d89b02c3f209 WatchSource:0}: Error finding container 03713469979a0430fe1577b7d9a43e1136c3b092e19ca2cdf8d1d89b02c3f209: Status 404 returned error can't find the container with id 03713469979a0430fe1577b7d9a43e1136c3b092e19ca2cdf8d1d89b02c3f209 Nov 02 21:58:15.318253 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:15.317317 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bzm9b" event={"ID":"9b08273d-5f57-44e7-896d-61ee487fad84","Type":"ContainerStarted","Data":"c2ce36a79df4099a47e6d502a34f365d76a88d0a508ac89a6b97d8ddc6fb578f"} Nov 02 21:58:15.318253 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:15.317377 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bzm9b" event={"ID":"9b08273d-5f57-44e7-896d-61ee487fad84","Type":"ContainerStarted","Data":"fb71e39313e12945899d2f7c92c0438c1df5be772490350ca8fcae0ddf295f0b"} Nov 02 21:58:15.324712 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:15.323021 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" event={"ID":"0a657025-9495-432b-a966-db8b41cae3df","Type":"ContainerStarted","Data":"3ffa272b9760c87fc94af4fdaf28742dc9d591c95624d51849375f3285e4f286"} Nov 02 21:58:15.324712 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:15.323083 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" event={"ID":"0a657025-9495-432b-a966-db8b41cae3df","Type":"ContainerStarted","Data":"6b52a1f4e3680e55ac21fb307b133d705bb7eb67d835b6946f67f6a05e4737e2"} Nov 02 21:58:15.324712 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:15.323113 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" event={"ID":"0a657025-9495-432b-a966-db8b41cae3df","Type":"ContainerStarted","Data":"e00c9a63fec107389b0a267bd653deb623c7ae9607abb88b3997cba591590dd9"} Nov 02 21:58:15.325858 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:15.325782 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" event={"ID":"c9ef1d8c-8213-455c-a988-925d8cdaa2b5","Type":"ContainerStarted","Data":"03713469979a0430fe1577b7d9a43e1136c3b092e19ca2cdf8d1d89b02c3f209"} Nov 02 21:58:15.341664 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:15.341618 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bzm9b" podStartSLOduration=2.819563901 podStartE2EDuration="4.341605466s" totalImagesPullingTime="1.43312512s" totalInitContainerRuntime="88.916445ms" isStatefulPod=true podCreationTimestamp="2026-11-02 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:15.340273382 +0000 UTC m=+23.278386601" watchObservedRunningTime="2026-11-02 21:58:15.341605466 +0000 UTC m=+23.279718683" Nov 02 21:58:15.360222 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:15.360168 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k6wjv" podStartSLOduration=2.447996829 podStartE2EDuration="4.360149598s" totalImagesPullingTime="1.912152769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:15.359700133 +0000 UTC m=+23.297813351" watchObservedRunningTime="2026-11-02 21:58:15.360149598 +0000 UTC m=+23.298262816" Nov 02 21:58:16.251422 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.250264 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-79998fd94f-dxckg"] Nov 02 21:58:16.270489 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.270455 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79998fd94f-dxckg"] Nov 02 21:58:16.270645 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.270570 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.275883 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.275858 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 21:58:16.276011 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.275946 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qk42s" Nov 02 21:58:16.276824 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.276801 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 21:58:16.291403 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.291380 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 21:58:16.291491 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.291414 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 21:58:16.291545 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.291488 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-pp5rl\"" Nov 02 21:58:16.291740 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.291719 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c35kprfbkiaea\"" Nov 02 21:58:16.337946 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.337524 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v6vw\" (UniqueName: \"kubernetes.io/projected/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-kube-api-access-8v6vw\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.337946 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.337568 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-secret-metrics-server-tls\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.337946 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.337599 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-metrics-server-audit-profiles\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.339064 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.338440 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-client-ca-bundle\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.339064 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.338489 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-audit-log\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.339064 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.338514 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.339064 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.338577 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-secret-metrics-server-client-certs\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.439684 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.439646 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v6vw\" (UniqueName: \"kubernetes.io/projected/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-kube-api-access-8v6vw\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.439954 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.439924 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-secret-metrics-server-tls\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.440061 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.439970 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-metrics-server-audit-profiles\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.440061 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.440022 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-client-ca-bundle\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.440061 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.440051 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-audit-log\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.440227 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.440088 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.440227 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.440178 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-secret-metrics-server-client-certs\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.441428 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.440820 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-audit-log\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.441428 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.441398 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.442296 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.442273 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-metrics-server-audit-profiles\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.443980 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.443933 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-secret-metrics-server-client-certs\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.443980 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.443970 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-secret-metrics-server-tls\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.444310 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.444279 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-client-ca-bundle\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.448986 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.448963 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v6vw\" (UniqueName: \"kubernetes.io/projected/9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3-kube-api-access-8v6vw\") pod \"metrics-server-79998fd94f-dxckg\" (UID: \"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3\") " pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.613640 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.613606 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:16.909950 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.909857 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79998fd94f-dxckg"] Nov 02 21:58:16.915098 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:58:16.915059 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efe0f4f_aa61_43a1_8c50_d2380a6fa2f3.slice/crio-c7deef1b67c0e8cd8f34d67ba6fadc5d02db948f3961ec157dbf24a34b920fa4 WatchSource:0}: Error finding container c7deef1b67c0e8cd8f34d67ba6fadc5d02db948f3961ec157dbf24a34b920fa4: Status 404 returned error can't find the container with id c7deef1b67c0e8cd8f34d67ba6fadc5d02db948f3961ec157dbf24a34b920fa4 Nov 02 21:58:16.949974 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:16.949949 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pgrkb" Nov 02 21:58:17.337583 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:17.337549 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerID="97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac" exitCode=0 Nov 02 21:58:17.337785 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:17.337645 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerDied","Data":"97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac"} Nov 02 21:58:17.339746 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:17.339710 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" event={"ID":"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3","Type":"ContainerStarted","Data":"c7deef1b67c0e8cd8f34d67ba6fadc5d02db948f3961ec157dbf24a34b920fa4"} Nov 02 21:58:18.230697 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.218225 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:58:18.261290 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.258085 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:58:18.261290 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.258281 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.265253 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.265148 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 21:58:18.289725 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.289398 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 21:58:18.289725 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.289669 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 21:58:18.291097 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.291074 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 21:58:18.291221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.291115 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 21:58:18.306623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.305153 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 21:58:18.306623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.305896 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 21:58:18.306623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.305957 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 21:58:18.306623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.306051 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 21:58:18.306623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.306146 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 21:58:18.306623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.306227 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 21:58:18.306623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.306325 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-552fk\"" Nov 02 21:58:18.306623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.306551 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 21:58:18.306623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.306585 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 21:58:18.307206 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.306654 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ein12ipsqq4i\"" Nov 02 21:58:18.307206 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.306817 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361374 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361428 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361456 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361494 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361516 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361534 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361555 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-web-config\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361605 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361629 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.361704 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361650 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361723 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361908 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361938 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361955 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.361991 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.362023 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config-out\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.362042 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.362067 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.362090 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.362723 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.362116 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vlqd\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-kube-api-access-4vlqd\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.383008 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.382966 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-z8gn7" event={"ID":"660abcf1-bcc1-486a-a0f1-dec089f5cc42","Type":"ContainerStarted","Data":"d0a41087a2935a2d62e6f5c6fd47dc0fa25f44d8e002e4531f6da6387344d7ed"} Nov 02 21:58:18.408614 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.408549 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:58:18.419578 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.419542 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-z8gn7" Nov 02 21:58:18.427588 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.427536 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-z8gn7" podStartSLOduration=2.152020597 podStartE2EDuration="20.427517315s" totalImagesPullingTime="18.275496718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:18.424944373 +0000 UTC m=+26.363057592" watchObservedRunningTime="2026-11-02 21:58:18.427517315 +0000 UTC m=+26.365630537" Nov 02 21:58:18.463620 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.463582 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.463837 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.463639 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.463837 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.463687 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.463837 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.463727 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.463837 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.463755 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.463837 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.463836 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.466340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.464401 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.466527 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466503 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config-out\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.466616 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466534 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.466616 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466560 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.466616 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466586 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.466616 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466610 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vlqd\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-kube-api-access-4vlqd\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.466865 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466642 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.466865 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466811 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.466865 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466842 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.467020 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466870 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.467020 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466908 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.467020 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466931 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.467020 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466957 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-web-config\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.467218 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.467047 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.471686 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.471353 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.471686 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.465303 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.471686 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.464546 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.473546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.473223 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.476689 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.474608 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.477739 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.477718 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.487577 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.487111 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.488502 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.488481 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.490067 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.489749 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.490839 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466312 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.490987 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.466219 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.491548 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.491511 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.523851 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.523778 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-web-config\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.524841 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.524812 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.525039 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.524853 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.525746 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.525458 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.525746 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.525702 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config-out\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.527567 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.527542 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.527745 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.527720 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.529229 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.529172 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vlqd\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-kube-api-access-4vlqd\") pod \"prometheus-k8s-0\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:18.786470 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:18.786149 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:19.174897 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:19.174860 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:58:19.179347 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:58:19.179302 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a24b4fc_8ecd_4f2b_946f_a78194105d94.slice/crio-fb334a594817477d63b621c3fbca84b479bd748c50ceb65531fc32ba6f317d57 WatchSource:0}: Error finding container fb334a594817477d63b621c3fbca84b479bd748c50ceb65531fc32ba6f317d57: Status 404 returned error can't find the container with id fb334a594817477d63b621c3fbca84b479bd748c50ceb65531fc32ba6f317d57 Nov 02 21:58:19.391386 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:19.391355 4309 generic.go:356] "Generic (PLEG): container finished" podID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerID="7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0" exitCode=0 Nov 02 21:58:19.392009 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:19.391979 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerDied","Data":"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0"} Nov 02 21:58:19.392231 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:19.392064 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerStarted","Data":"fb334a594817477d63b621c3fbca84b479bd748c50ceb65531fc32ba6f317d57"} Nov 02 21:58:20.399146 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:20.399067 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" event={"ID":"c9ef1d8c-8213-455c-a988-925d8cdaa2b5","Type":"ContainerStarted","Data":"838e44fb26311444bf35ddd9486da01e53bd7f9e22a82cf3b7e534a0347477eb"} Nov 02 21:58:20.399587 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:20.399163 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" event={"ID":"c9ef1d8c-8213-455c-a988-925d8cdaa2b5","Type":"ContainerStarted","Data":"1abfa2d4f8ba04d1f5420458fd3a622fabbd69be8eb3897b416b0b6cba6a444e"} Nov 02 21:58:20.399587 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:20.399212 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" event={"ID":"c9ef1d8c-8213-455c-a988-925d8cdaa2b5","Type":"ContainerStarted","Data":"7675e1a72922b79948bad6534153fa5f8eb5a0af4c6b7a8b1c62a3f8322ff7d2"} Nov 02 21:58:20.402817 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:20.402720 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" event={"ID":"9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3","Type":"ContainerStarted","Data":"183d2fde7b9ca2e2e0fabe80557620777c58d85fda72fa39bf4ddc362ba540f0"} Nov 02 21:58:20.405522 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:20.405494 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7cbb8bcd-hccnd" event={"ID":"391f0200-66e0-46a2-90c5-e34878a8b4af","Type":"ContainerStarted","Data":"de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc"} Nov 02 21:58:20.422355 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:20.422128 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" podStartSLOduration=1.196558514 podStartE2EDuration="4.422109286s" totalImagesPullingTime="3.225550772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:20.421883911 +0000 UTC m=+28.359997133" watchObservedRunningTime="2026-11-02 21:58:20.422109286 +0000 UTC m=+28.360222502" Nov 02 21:58:20.441811 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:20.441756 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f7cbb8bcd-hccnd" podStartSLOduration=2.022832937 podStartE2EDuration="7.441736746s" totalImagesPullingTime="5.418903809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:20.440981749 +0000 UTC m=+28.379094969" watchObservedRunningTime="2026-11-02 21:58:20.441736746 +0000 UTC m=+28.379849967" Nov 02 21:58:22.417571 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:22.417534 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" event={"ID":"c9ef1d8c-8213-455c-a988-925d8cdaa2b5","Type":"ContainerStarted","Data":"7dfeb4868a393a7f20f603628ee98bd1656adb011f97ad19c3491039ff19949f"} Nov 02 21:58:22.418064 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:22.417597 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" event={"ID":"c9ef1d8c-8213-455c-a988-925d8cdaa2b5","Type":"ContainerStarted","Data":"0f6516c383e0144b3734a0cb452c3ccb90b4ba7be54abacf9d85ee95e6a2905d"} Nov 02 21:58:22.418064 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:22.417624 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" event={"ID":"c9ef1d8c-8213-455c-a988-925d8cdaa2b5","Type":"ContainerStarted","Data":"8cedc463bd9895bc45e2d13818f3187d82b619b4e9e9147f19bc3553855beadd"} Nov 02 21:58:22.418064 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:22.417698 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:22.442984 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:22.442922 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" podStartSLOduration=2.629999637 podStartE2EDuration="9.442899858s" totalImagesPullingTime="6.812900221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:22.441235674 +0000 UTC m=+30.379348893" watchObservedRunningTime="2026-11-02 21:58:22.442899858 +0000 UTC m=+30.381013078" Nov 02 21:58:23.644585 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:23.643827 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:23.644585 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:23.643936 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:23.652590 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:23.652490 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:24.431569 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:24.431488 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerStarted","Data":"0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5"} Nov 02 21:58:24.431569 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:24.431563 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerStarted","Data":"05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b"} Nov 02 21:58:24.431879 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:24.431591 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerStarted","Data":"0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6"} Nov 02 21:58:24.431879 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:24.431615 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerStarted","Data":"d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7"} Nov 02 21:58:24.431879 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:24.431647 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerStarted","Data":"15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb"} Nov 02 21:58:24.431879 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:24.431708 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerStarted","Data":"b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5"} Nov 02 21:58:24.437404 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:24.437360 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:24.459321 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:24.458593 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.959941901 podStartE2EDuration="12.458574415s" totalImagesPullingTime="8.477206431s" totalInitContainerRuntime="21.426083ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:24.456252416 +0000 UTC m=+32.394365635" watchObservedRunningTime="2026-11-02 21:58:24.458574415 +0000 UTC m=+32.396687636" Nov 02 21:58:27.448209 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:27.448178 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerStarted","Data":"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3"} Nov 02 21:58:27.448775 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:27.448241 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerStarted","Data":"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a"} Nov 02 21:58:27.448775 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:27.448269 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerStarted","Data":"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6"} Nov 02 21:58:27.448775 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:27.448295 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerStarted","Data":"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3"} Nov 02 21:58:27.448775 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:27.448327 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerStarted","Data":"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d"} Nov 02 21:58:27.448775 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:27.448352 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerStarted","Data":"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea"} Nov 02 21:58:27.469348 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:27.469325 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pft59" Nov 02 21:58:27.475962 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:27.475891 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.318771633 podStartE2EDuration="9.475871308s" totalImagesPullingTime="7.140501736s" totalInitContainerRuntime="16.597939ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:58:27.473605289 +0000 UTC m=+35.411718508" watchObservedRunningTime="2026-11-02 21:58:27.475871308 +0000 UTC m=+35.413984528" Nov 02 21:58:28.428496 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:28.428467 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" Nov 02 21:58:28.787342 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:28.787263 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:33.229786 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:33.229751 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f7cbb8bcd-hccnd"] Nov 02 21:58:36.614564 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:36.614480 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:36.615024 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:36.614574 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:41.593652 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:58:41.593617 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 21:58:52.419014 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:52.418983 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:52.485299 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:52.485274 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:52.560967 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:52.560945 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:58:56.619037 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:56.619007 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:56.622884 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:56.622863 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-79998fd94f-dxckg" Nov 02 21:58:58.251295 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.251251 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f7cbb8bcd-hccnd" podUID="391f0200-66e0-46a2-90c5-e34878a8b4af" containerName="console" containerID="cri-o://de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc" gracePeriod=15 Nov 02 21:58:58.529798 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.529767 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:58.569446 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.569410 4309 generic.go:356] "Generic (PLEG): container finished" podID="391f0200-66e0-46a2-90c5-e34878a8b4af" containerID="de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc" exitCode=0 Nov 02 21:58:58.569622 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.569460 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7cbb8bcd-hccnd" event={"ID":"391f0200-66e0-46a2-90c5-e34878a8b4af","Type":"ContainerDied","Data":"de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc"} Nov 02 21:58:58.569622 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.569470 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7cbb8bcd-hccnd" Nov 02 21:58:58.569622 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.569543 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7cbb8bcd-hccnd" event={"ID":"391f0200-66e0-46a2-90c5-e34878a8b4af","Type":"ContainerDied","Data":"3eeb1101b14cff587f026b0bc2eeb898afba3e033d39be01a81665b2f29464c5"} Nov 02 21:58:58.569622 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.569578 4309 scope.go:122] "RemoveContainer" containerID="de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc" Nov 02 21:58:58.577872 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.577855 4309 scope.go:122] "RemoveContainer" containerID="de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc" Nov 02 21:58:58.578141 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:58:58.578121 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc\": container with ID starting with de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc not found: ID does not exist" containerID="de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc" Nov 02 21:58:58.578208 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.578151 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc"} err="failed to get container status \"de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc\": rpc error: code = NotFound desc = could not find container \"de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc\": container with ID starting with de4688bd0564140d64c02b6fb4a9f3a0fde4c250d2b5a51b42d1106229d203dc not found: ID does not exist" Nov 02 21:58:58.629410 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629383 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391f0200-66e0-46a2-90c5-e34878a8b4af-tmp\" (UniqueName: \"kubernetes.io/empty-dir/391f0200-66e0-46a2-90c5-e34878a8b4af-tmp\") pod \"391f0200-66e0-46a2-90c5-e34878a8b4af\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " Nov 02 21:58:58.629573 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629420 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-service-ca\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-service-ca\") pod \"391f0200-66e0-46a2-90c5-e34878a8b4af\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " Nov 02 21:58:58.629573 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629471 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391f0200-66e0-46a2-90c5-e34878a8b4af-kube-api-access-cmmps\" (UniqueName: \"kubernetes.io/projected/391f0200-66e0-46a2-90c5-e34878a8b4af-kube-api-access-cmmps\") pod \"391f0200-66e0-46a2-90c5-e34878a8b4af\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " Nov 02 21:58:58.629573 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629495 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-console-config\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-console-config\") pod \"391f0200-66e0-46a2-90c5-e34878a8b4af\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " Nov 02 21:58:58.629573 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629526 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-oauth-config\") pod \"391f0200-66e0-46a2-90c5-e34878a8b4af\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " Nov 02 21:58:58.629573 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629558 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-serving-cert\") pod \"391f0200-66e0-46a2-90c5-e34878a8b4af\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " Nov 02 21:58:58.629858 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629590 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-oauth-serving-cert\") pod \"391f0200-66e0-46a2-90c5-e34878a8b4af\" (UID: \"391f0200-66e0-46a2-90c5-e34878a8b4af\") " Nov 02 21:58:58.629858 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629739 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391f0200-66e0-46a2-90c5-e34878a8b4af-tmp" pod "391f0200-66e0-46a2-90c5-e34878a8b4af" (UID: "391f0200-66e0-46a2-90c5-e34878a8b4af"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:58:58.629967 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629885 4309 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/391f0200-66e0-46a2-90c5-e34878a8b4af-tmp\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:58.630026 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.629987 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-console-config" pod "391f0200-66e0-46a2-90c5-e34878a8b4af" (UID: "391f0200-66e0-46a2-90c5-e34878a8b4af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:58.630081 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.630050 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-oauth-serving-cert" pod "391f0200-66e0-46a2-90c5-e34878a8b4af" (UID: "391f0200-66e0-46a2-90c5-e34878a8b4af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:58.630125 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.630071 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-service-ca" pod "391f0200-66e0-46a2-90c5-e34878a8b4af" (UID: "391f0200-66e0-46a2-90c5-e34878a8b4af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:58:58.631891 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.631864 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391f0200-66e0-46a2-90c5-e34878a8b4af-kube-api-access-cmmps" pod "391f0200-66e0-46a2-90c5-e34878a8b4af" (UID: "391f0200-66e0-46a2-90c5-e34878a8b4af"). InnerVolumeSpecName "kube-api-access-cmmps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:58:58.631992 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.631902 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-serving-cert" pod "391f0200-66e0-46a2-90c5-e34878a8b4af" (UID: "391f0200-66e0-46a2-90c5-e34878a8b4af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:58:58.631992 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.631927 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-oauth-config" pod "391f0200-66e0-46a2-90c5-e34878a8b4af" (UID: "391f0200-66e0-46a2-90c5-e34878a8b4af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:58:58.730692 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.730654 4309 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-service-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:58.730692 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.730698 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmmps\" (UniqueName: \"kubernetes.io/projected/391f0200-66e0-46a2-90c5-e34878a8b4af-kube-api-access-cmmps\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:58.730888 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.730709 4309 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-console-config\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:58.730888 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.730718 4309 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-oauth-config\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:58.730888 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.730726 4309 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/391f0200-66e0-46a2-90c5-e34878a8b4af-console-serving-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:58.730888 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.730735 4309 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/391f0200-66e0-46a2-90c5-e34878a8b4af-oauth-serving-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:58:58.903818 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.891139 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f7cbb8bcd-hccnd"] Nov 02 21:58:58.903818 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:58:58.894462 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f7cbb8bcd-hccnd"] Nov 02 21:59:00.185646 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:00.185614 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391f0200-66e0-46a2-90c5-e34878a8b4af" path="/var/lib/kubelet/pods/391f0200-66e0-46a2-90c5-e34878a8b4af/volumes" Nov 02 21:59:02.123174 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.123131 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:59:02.123711 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.123473 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="alertmanager" containerID="cri-o://b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5" gracePeriod=120 Nov 02 21:59:02.123711 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.123530 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy-metric" containerID="cri-o://05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b" gracePeriod=120 Nov 02 21:59:02.123711 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.123561 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="prom-label-proxy" containerID="cri-o://0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5" gracePeriod=120 Nov 02 21:59:02.123711 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.123600 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="config-reloader" containerID="cri-o://15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb" gracePeriod=120 Nov 02 21:59:02.123711 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.123631 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy" containerID="cri-o://0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6" gracePeriod=120 Nov 02 21:59:02.123711 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.123586 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy-web" containerID="cri-o://d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7" gracePeriod=120 Nov 02 21:59:02.588086 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588048 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerID="0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5" exitCode=0 Nov 02 21:59:02.588086 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588074 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerID="05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b" exitCode=0 Nov 02 21:59:02.588086 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588080 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerID="0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6" exitCode=0 Nov 02 21:59:02.588086 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588086 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerID="15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb" exitCode=0 Nov 02 21:59:02.588086 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588095 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerID="b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5" exitCode=0 Nov 02 21:59:02.588353 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588118 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerDied","Data":"0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5"} Nov 02 21:59:02.588353 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588203 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerDied","Data":"05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b"} Nov 02 21:59:02.588353 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588228 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerDied","Data":"0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6"} Nov 02 21:59:02.588353 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588252 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerDied","Data":"15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb"} Nov 02 21:59:02.588353 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:02.588274 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerDied","Data":"b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5"} Nov 02 21:59:03.390117 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.390091 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.469788 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.469706 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-main-db\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.469788 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.469753 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-kube-api-access-2sjlp\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-kube-api-access-2sjlp\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470015 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.469785 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470015 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.469824 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-web\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470015 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.469854 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-metrics-client-ca\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470015 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.469895 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-out\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470015 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.469937 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-volume\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-volume\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470015 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.469962 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470397 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470024 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-main-db" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:03.470397 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470269 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:03.470397 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470331 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-metrics-client-ca" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:03.470397 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470384 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-cluster-tls-config\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470607 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470466 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-web-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-web-config\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470607 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470514 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.470607 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470546 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-main-tls\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.471194 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470621 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-tls-assets\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-tls-assets\") pod \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\" (UID: \"f2879ff3-1cff-4772-8d5c-1c2818edcfef\") " Nov 02 21:59:03.471194 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470875 4309 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-main-db\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.471194 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470894 4309 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.471194 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.470910 4309 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2879ff3-1cff-4772-8d5c-1c2818edcfef-metrics-client-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.472758 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.472728 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-kube-api-access-2sjlp" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "kube-api-access-2sjlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:03.473278 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.473251 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-web" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:03.473660 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.473545 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:03.474106 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.473958 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-volume" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:03.474106 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.474069 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-tls-assets" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:03.474536 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.474501 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-out" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:03.474829 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.474792 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-metric" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:03.474933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.474839 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-main-tls" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:03.478474 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.478449 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-cluster-tls-config" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:03.484697 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.484655 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-web-config" pod "f2879ff3-1cff-4772-8d5c-1c2818edcfef" (UID: "f2879ff3-1cff-4772-8d5c-1c2818edcfef"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:03.571970 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.571936 4309 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-tls-assets\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.571970 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.571965 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjlp\" (UniqueName: \"kubernetes.io/projected/f2879ff3-1cff-4772-8d5c-1c2818edcfef-kube-api-access-2sjlp\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.571970 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.571978 4309 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.572210 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.571988 4309 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-out\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.572210 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.571997 4309 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.572210 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.572006 4309 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.572210 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.572016 4309 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-cluster-tls-config\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.572210 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.572026 4309 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-web-config\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.572210 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.572034 4309 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.572210 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.572042 4309 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f2879ff3-1cff-4772-8d5c-1c2818edcfef-secret-alertmanager-main-tls\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:03.597952 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.597926 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerID="d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7" exitCode=0 Nov 02 21:59:03.598113 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.597981 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerDied","Data":"d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7"} Nov 02 21:59:03.598113 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.598002 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.598113 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.598032 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f2879ff3-1cff-4772-8d5c-1c2818edcfef","Type":"ContainerDied","Data":"074b2e5852bd9b95deee6ad6138d6e9eb9e37886eb4ffb1aab18989408a2a812"} Nov 02 21:59:03.598113 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.598063 4309 scope.go:122] "RemoveContainer" containerID="0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5" Nov 02 21:59:03.605944 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.605804 4309 scope.go:122] "RemoveContainer" containerID="05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b" Nov 02 21:59:03.613380 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.613358 4309 scope.go:122] "RemoveContainer" containerID="0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6" Nov 02 21:59:03.620004 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.619975 4309 scope.go:122] "RemoveContainer" containerID="d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7" Nov 02 21:59:03.622641 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.622619 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:59:03.625644 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.625622 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:59:03.629304 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.629276 4309 scope.go:122] "RemoveContainer" containerID="15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb" Nov 02 21:59:03.636216 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.636200 4309 scope.go:122] "RemoveContainer" containerID="b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5" Nov 02 21:59:03.643258 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.643243 4309 scope.go:122] "RemoveContainer" containerID="97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac" Nov 02 21:59:03.650541 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650521 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:59:03.650811 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650792 4309 scope.go:122] "RemoveContainer" containerID="0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5" Nov 02 21:59:03.650926 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650910 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="config-reloader" Nov 02 21:59:03.650998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650928 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="config-reloader" Nov 02 21:59:03.650998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650942 4309 cpu_manager.go:414] "containerMap: removing container" podUID="391f0200-66e0-46a2-90c5-e34878a8b4af" containerName="console" Nov 02 21:59:03.650998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650950 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391f0200-66e0-46a2-90c5-e34878a8b4af" containerName="console" Nov 02 21:59:03.650998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650958 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="alertmanager" Nov 02 21:59:03.650998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650966 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="alertmanager" Nov 02 21:59:03.650998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650985 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="init-config-reloader" Nov 02 21:59:03.650998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.650992 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="init-config-reloader" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651007 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy-web" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651011 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy-web" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651019 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651024 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651030 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy-metric" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651035 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy-metric" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651041 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="prom-label-proxy" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651047 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="prom-label-proxy" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651096 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy-web" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651104 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651110 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="kube-rbac-proxy-metric" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651121 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="prom-label-proxy" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651129 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="config-reloader" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651135 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="391f0200-66e0-46a2-90c5-e34878a8b4af" containerName="console" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651140 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" containerName="alertmanager" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:03.651103 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5\": container with ID starting with 0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5 not found: ID does not exist" containerID="0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651206 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5"} err="failed to get container status \"0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5\": rpc error: code = NotFound desc = could not find container \"0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5\": container with ID starting with 0053be76af50dd0e9d15d8c7849806947902530cd31f85f39889e6b4ae9bc4a5 not found: ID does not exist" Nov 02 21:59:03.651221 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651222 4309 scope.go:122] "RemoveContainer" containerID="05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b" Nov 02 21:59:03.652007 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:03.651543 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b\": container with ID starting with 05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b not found: ID does not exist" containerID="05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b" Nov 02 21:59:03.652007 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651570 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b"} err="failed to get container status \"05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b\": rpc error: code = NotFound desc = could not find container \"05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b\": container with ID starting with 05b9d14aa62bb351be42bfc2674f3bade0869359938c09f82d87bca1d86da83b not found: ID does not exist" Nov 02 21:59:03.652007 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.651593 4309 scope.go:122] "RemoveContainer" containerID="0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6" Nov 02 21:59:03.652329 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:03.652304 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6\": container with ID starting with 0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6 not found: ID does not exist" containerID="0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6" Nov 02 21:59:03.652414 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.652338 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6"} err="failed to get container status \"0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6\": rpc error: code = NotFound desc = could not find container \"0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6\": container with ID starting with 0a6e119c20c8996f6e0556d77bc58fe3dd8f1b9b0c3de1aaf8aae953a159bbc6 not found: ID does not exist" Nov 02 21:59:03.652414 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.652360 4309 scope.go:122] "RemoveContainer" containerID="d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7" Nov 02 21:59:03.652647 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:03.652629 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7\": container with ID starting with d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7 not found: ID does not exist" containerID="d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7" Nov 02 21:59:03.652765 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.652651 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7"} err="failed to get container status \"d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7\": rpc error: code = NotFound desc = could not find container \"d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7\": container with ID starting with d8c0b1d93b070f4b7d55c35e0fb4f6f9afadb10634ab091f3019f3c65e7141b7 not found: ID does not exist" Nov 02 21:59:03.652765 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.652669 4309 scope.go:122] "RemoveContainer" containerID="15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb" Nov 02 21:59:03.652966 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:03.652947 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb\": container with ID starting with 15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb not found: ID does not exist" containerID="15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb" Nov 02 21:59:03.653011 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.652970 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb"} err="failed to get container status \"15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb\": rpc error: code = NotFound desc = could not find container \"15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb\": container with ID starting with 15a25bdd0e1d2367044c64bbc1077d99255d3ff5f982194c9b91507fbe3a6acb not found: ID does not exist" Nov 02 21:59:03.653011 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.652984 4309 scope.go:122] "RemoveContainer" containerID="b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5" Nov 02 21:59:03.653234 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:03.653217 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5\": container with ID starting with b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5 not found: ID does not exist" containerID="b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5" Nov 02 21:59:03.653282 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.653239 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5"} err="failed to get container status \"b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5\": rpc error: code = NotFound desc = could not find container \"b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5\": container with ID starting with b96e3abc790a892ae4203e7b8689fb449e2665cd6e8c6ccb5b52abdb1e4c29b5 not found: ID does not exist" Nov 02 21:59:03.653282 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.653254 4309 scope.go:122] "RemoveContainer" containerID="97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac" Nov 02 21:59:03.653470 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:03.653454 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac\": container with ID starting with 97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac not found: ID does not exist" containerID="97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac" Nov 02 21:59:03.653525 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.653478 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac"} err="failed to get container status \"97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac\": rpc error: code = NotFound desc = could not find container \"97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac\": container with ID starting with 97008b1595b4f4d134c1e5ace7edebb219176f692e97848e53acc6d3fc00ebac not found: ID does not exist" Nov 02 21:59:03.677363 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.677332 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:59:03.677537 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.677464 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.681687 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.681580 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 21:59:03.681687 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.681617 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 21:59:03.681687 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.681629 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 21:59:03.681994 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.681729 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 21:59:03.681994 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.681763 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 21:59:03.681994 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.681765 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 21:59:03.681994 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.681776 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 21:59:03.681994 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.681799 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 21:59:03.682884 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.682870 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n2nmp\"" Nov 02 21:59:03.684846 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.684824 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 21:59:03.773716 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.773622 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-web-config\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.773716 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.773656 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1928586d-c241-4777-ba20-d31b032dec7e-config-out\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.773716 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.773700 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1928586d-c241-4777-ba20-d31b032dec7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.773933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.773782 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.773933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.773812 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtq4\" (UniqueName: \"kubernetes.io/projected/1928586d-c241-4777-ba20-d31b032dec7e-kube-api-access-nqtq4\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.773933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.773832 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1928586d-c241-4777-ba20-d31b032dec7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.773933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.773878 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.774048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.773960 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1928586d-c241-4777-ba20-d31b032dec7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.774048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.773989 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.774048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.774014 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.774048 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.774036 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.774163 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.774083 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.774163 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.774104 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1928586d-c241-4777-ba20-d31b032dec7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.874899 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.874864 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.874899 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.874903 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1928586d-c241-4777-ba20-d31b032dec7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875151 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.874930 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875151 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.874957 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875151 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.874978 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875151 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.874993 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875151 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.875009 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1928586d-c241-4777-ba20-d31b032dec7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875151 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.875035 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-web-config\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875456 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.875392 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1928586d-c241-4777-ba20-d31b032dec7e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875549 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.875527 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1928586d-c241-4777-ba20-d31b032dec7e-config-out\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875614 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.875579 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1928586d-c241-4777-ba20-d31b032dec7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875690 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.875626 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875690 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.875666 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtq4\" (UniqueName: \"kubernetes.io/projected/1928586d-c241-4777-ba20-d31b032dec7e-kube-api-access-nqtq4\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875816 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.875716 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1928586d-c241-4777-ba20-d31b032dec7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.875870 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.875823 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1928586d-c241-4777-ba20-d31b032dec7e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.876632 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.876602 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1928586d-c241-4777-ba20-d31b032dec7e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.878187 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.878137 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-config-volume\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.878305 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.878227 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.878577 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.878558 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.878629 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.878568 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.878629 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.878583 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-web-config\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.878817 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.878797 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1928586d-c241-4777-ba20-d31b032dec7e-config-out\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.879025 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.879004 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.879025 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.879015 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1928586d-c241-4777-ba20-d31b032dec7e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.879747 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.879732 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1928586d-c241-4777-ba20-d31b032dec7e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:03.883578 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:03.883558 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtq4\" (UniqueName: \"kubernetes.io/projected/1928586d-c241-4777-ba20-d31b032dec7e-kube-api-access-nqtq4\") pod \"alertmanager-main-0\" (UID: \"1928586d-c241-4777-ba20-d31b032dec7e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:04.088077 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:04.088041 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 21:59:04.189164 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:04.187289 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2879ff3-1cff-4772-8d5c-1c2818edcfef" path="/var/lib/kubelet/pods/f2879ff3-1cff-4772-8d5c-1c2818edcfef/volumes" Nov 02 21:59:04.390457 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:04.390434 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 21:59:04.392048 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:59:04.392024 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1928586d_c241_4777_ba20_d31b032dec7e.slice/crio-fac483f28f361d850d676228b61020785e4d4f1a707c79a3a8be555d0cd5705f WatchSource:0}: Error finding container fac483f28f361d850d676228b61020785e4d4f1a707c79a3a8be555d0cd5705f: Status 404 returned error can't find the container with id fac483f28f361d850d676228b61020785e4d4f1a707c79a3a8be555d0cd5705f Nov 02 21:59:04.603457 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:04.603418 4309 generic.go:356] "Generic (PLEG): container finished" podID="1928586d-c241-4777-ba20-d31b032dec7e" containerID="7d30faeba432ebfa5764f8213b3d8cc10d511df1c5895e4fb65bd017751dbf0e" exitCode=0 Nov 02 21:59:04.603631 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:04.603514 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1928586d-c241-4777-ba20-d31b032dec7e","Type":"ContainerDied","Data":"7d30faeba432ebfa5764f8213b3d8cc10d511df1c5895e4fb65bd017751dbf0e"} Nov 02 21:59:04.603631 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:04.603587 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1928586d-c241-4777-ba20-d31b032dec7e","Type":"ContainerStarted","Data":"fac483f28f361d850d676228b61020785e4d4f1a707c79a3a8be555d0cd5705f"} Nov 02 21:59:05.611556 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:05.611524 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1928586d-c241-4777-ba20-d31b032dec7e","Type":"ContainerStarted","Data":"8210793c4ee3116567828e42c1e3021888bfdbfb4a609123f9925f455ee5d3ac"} Nov 02 21:59:05.611933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:05.611586 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1928586d-c241-4777-ba20-d31b032dec7e","Type":"ContainerStarted","Data":"b9d23c4f3766f8d1fe042f3d4c34c503cb27ef6efedc988ee4da306f5980f121"} Nov 02 21:59:05.611933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:05.611613 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1928586d-c241-4777-ba20-d31b032dec7e","Type":"ContainerStarted","Data":"f5e650424966ec653c2232199dec493ea2d84320e8afdb736ed392c4d24bb51c"} Nov 02 21:59:05.611933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:05.611642 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1928586d-c241-4777-ba20-d31b032dec7e","Type":"ContainerStarted","Data":"34289822d296a9288b7bcb0f40bb6c9fffb791dcdfc872d944a28b5331d1c14e"} Nov 02 21:59:05.611933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:05.611668 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1928586d-c241-4777-ba20-d31b032dec7e","Type":"ContainerStarted","Data":"528d0abc6e71543e1ba1bc97115d804dbffb2dff4a240b2c8fa821c9ded98c85"} Nov 02 21:59:05.611933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:05.611707 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1928586d-c241-4777-ba20-d31b032dec7e","Type":"ContainerStarted","Data":"8b96c2f30a7bf77cfebfa4dc2b6f4418f402764fa3f6a36f21203c188a448104"} Nov 02 21:59:05.637846 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:05.637803 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.623471298 podStartE2EDuration="2.63778568s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.314382ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:05.635394151 +0000 UTC m=+73.573507370" watchObservedRunningTime="2026-11-02 21:59:05.63778568 +0000 UTC m=+73.575898904" Nov 02 21:59:06.131443 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.131406 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7dc44596d8-r9jct"] Nov 02 21:59:06.152765 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.152735 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dc44596d8-r9jct"] Nov 02 21:59:06.152919 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.152861 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.163111 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.163086 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 21:59:06.170087 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.170055 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 21:59:06.196597 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.196560 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-secret-telemeter-client\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.196597 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.196597 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-telemeter-client-tls\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.196847 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.196736 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6022facf-3f59-4c38-bcb1-d7a8e3853e68-serving-certs-ca-bundle\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.196847 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.196775 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-federate-client-tls\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.196847 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.196797 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6022facf-3f59-4c38-bcb1-d7a8e3853e68-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.196967 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.196866 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.196967 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.196895 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6022facf-3f59-4c38-bcb1-d7a8e3853e68-metrics-client-ca\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.197030 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.196987 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfd6\" (UniqueName: \"kubernetes.io/projected/6022facf-3f59-4c38-bcb1-d7a8e3853e68-kube-api-access-grfd6\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.224746 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.224722 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 21:59:06.224746 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.224729 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 21:59:06.224900 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.224738 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hj6zw\"" Nov 02 21:59:06.224900 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.224740 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 21:59:06.224900 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.224773 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 21:59:06.297632 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.297600 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfd6\" (UniqueName: \"kubernetes.io/projected/6022facf-3f59-4c38-bcb1-d7a8e3853e68-kube-api-access-grfd6\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.297795 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.297649 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-secret-telemeter-client\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.297795 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.297669 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-telemeter-client-tls\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.297795 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.297737 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6022facf-3f59-4c38-bcb1-d7a8e3853e68-serving-certs-ca-bundle\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.297795 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.297761 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-federate-client-tls\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.297795 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.297782 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6022facf-3f59-4c38-bcb1-d7a8e3853e68-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.298029 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.297819 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.298029 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.297844 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6022facf-3f59-4c38-bcb1-d7a8e3853e68-metrics-client-ca\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.298697 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.298643 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6022facf-3f59-4c38-bcb1-d7a8e3853e68-serving-certs-ca-bundle\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.298821 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.298779 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6022facf-3f59-4c38-bcb1-d7a8e3853e68-metrics-client-ca\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.298897 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.298832 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6022facf-3f59-4c38-bcb1-d7a8e3853e68-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.300426 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.300400 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-secret-telemeter-client\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.300531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.300437 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-telemeter-client-tls\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.300531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.300454 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-federate-client-tls\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.300531 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.300461 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6022facf-3f59-4c38-bcb1-d7a8e3853e68-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.306802 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.306782 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfd6\" (UniqueName: \"kubernetes.io/projected/6022facf-3f59-4c38-bcb1-d7a8e3853e68-kube-api-access-grfd6\") pod \"telemeter-client-7dc44596d8-r9jct\" (UID: \"6022facf-3f59-4c38-bcb1-d7a8e3853e68\") " pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.506999 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.506920 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:59:06.507246 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.507216 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="prometheus" containerID="cri-o://72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" gracePeriod=600 Nov 02 21:59:06.507377 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.507235 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy" containerID="cri-o://dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" gracePeriod=600 Nov 02 21:59:06.507377 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.507263 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy-thanos" containerID="cri-o://38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" gracePeriod=600 Nov 02 21:59:06.507377 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.507277 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="thanos-sidecar" containerID="cri-o://013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" gracePeriod=600 Nov 02 21:59:06.507562 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.507292 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="config-reloader" containerID="cri-o://93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" gracePeriod=600 Nov 02 21:59:06.507562 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.507378 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy-web" containerID="cri-o://af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" gracePeriod=600 Nov 02 21:59:06.531479 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.531454 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" Nov 02 21:59:06.808894 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.808588 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:06.836500 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.836469 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dc44596d8-r9jct"] Nov 02 21:59:06.840656 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:59:06.840632 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6022facf_3f59_4c38_bcb1_d7a8e3853e68.slice/crio-c7129b9abbf2a129adf14a2b1b10c84689bbf94a1417752a6c7445a09eaf9cb3 WatchSource:0}: Error finding container c7129b9abbf2a129adf14a2b1b10c84689bbf94a1417752a6c7445a09eaf9cb3: Status 404 returned error can't find the container with id c7129b9abbf2a129adf14a2b1b10c84689bbf94a1417752a6c7445a09eaf9cb3 Nov 02 21:59:06.904890 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.904862 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-2\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905073 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.904898 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905073 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.904924 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config-out\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905073 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.904948 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-metrics-client-certs\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905073 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.904968 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-1\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905073 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.904995 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-web-config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-web-config\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905073 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905028 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-kube-rbac-proxy\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905212 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-kubelet-serving-ca-bundle\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905245 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-2" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:06.905388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905265 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-trusted-ca-bundle\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905388 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905377 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905611 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905414 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-kube-api-access-4vlqd\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-kube-api-access-4vlqd\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905611 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905446 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-tls-assets\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-tls-assets\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905611 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905420 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-1" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:06.905611 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905490 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905611 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905533 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-tls\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905611 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905576 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-metrics-client-ca\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905965 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905627 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-db\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905965 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905659 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-0\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905965 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905721 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-serving-certs-ca-bundle\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905965 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905739 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-trusted-ca-bundle" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:06.905965 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905751 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-grpc-tls\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905965 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905790 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-thanos-prometheus-http-client-file\") pod \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\" (UID: \"7a24b4fc-8ecd-4f2b-946f-a78194105d94\") " Nov 02 21:59:06.905965 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.905865 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-kubelet-serving-ca-bundle" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:06.906308 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.906095 4309 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:06.906308 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.906118 4309 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:06.906308 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.906135 4309 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:06.906308 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.906152 4309 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:06.906994 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.906969 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-metrics-client-ca" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:06.908141 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.908095 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-serving-certs-ca-bundle" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:06.908259 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.908184 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-0" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:59:06.908382 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.908352 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-metrics-client-certs" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:06.908789 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.908759 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-tls-assets" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:06.908971 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.908951 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-db" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:06.909088 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.909067 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:06.909522 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.909430 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-tls" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:06.909725 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.909669 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-thanos-sidecar-tls" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:06.909978 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.909953 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-thanos-prometheus-http-client-file" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:06.910075 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.909989 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-kube-rbac-proxy" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:06.910075 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.910053 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-kube-rbac-proxy-web" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:06.910192 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.910086 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config-out" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 21:59:06.910565 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.910534 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-kube-api-access-4vlqd" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "kube-api-access-4vlqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:59:06.911074 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.911057 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-grpc-tls" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:06.919016 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:06.918996 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-web-config" pod "7a24b4fc-8ecd-4f2b-946f-a78194105d94" (UID: "7a24b4fc-8ecd-4f2b-946f-a78194105d94"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 21:59:07.007150 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007122 4309 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007150 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007147 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vlqd\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-kube-api-access-4vlqd\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007150 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007156 4309 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7a24b4fc-8ecd-4f2b-946f-a78194105d94-tls-assets\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007167 4309 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007177 4309 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007185 4309 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-metrics-client-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007193 4309 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-db\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007202 4309 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007211 4309 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a24b4fc-8ecd-4f2b-946f-a78194105d94-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007219 4309 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-grpc-tls\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007227 4309 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007237 4309 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007246 4309 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7a24b4fc-8ecd-4f2b-946f-a78194105d94-config-out\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007255 4309 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-metrics-client-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007263 4309 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-web-config\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.007340 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.007271 4309 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a24b4fc-8ecd-4f2b-946f-a78194105d94-secret-kube-rbac-proxy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 21:59:07.624125 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624095 4309 generic.go:356] "Generic (PLEG): container finished" podID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerID="38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" exitCode=0 Nov 02 21:59:07.624125 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624119 4309 generic.go:356] "Generic (PLEG): container finished" podID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerID="dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" exitCode=0 Nov 02 21:59:07.624125 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624125 4309 generic.go:356] "Generic (PLEG): container finished" podID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerID="af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" exitCode=0 Nov 02 21:59:07.624125 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624132 4309 generic.go:356] "Generic (PLEG): container finished" podID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerID="013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" exitCode=0 Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624138 4309 generic.go:356] "Generic (PLEG): container finished" podID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerID="93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" exitCode=0 Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624143 4309 generic.go:356] "Generic (PLEG): container finished" podID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerID="72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" exitCode=0 Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624169 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624172 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerDied","Data":"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3"} Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624255 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerDied","Data":"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a"} Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624294 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerDied","Data":"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6"} Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624331 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerDied","Data":"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3"} Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624339 4309 scope.go:122] "RemoveContainer" containerID="38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624375 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerDied","Data":"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d"} Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624417 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerDied","Data":"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea"} Nov 02 21:59:07.624453 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.624456 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7a24b4fc-8ecd-4f2b-946f-a78194105d94","Type":"ContainerDied","Data":"fb334a594817477d63b621c3fbca84b479bd748c50ceb65531fc32ba6f317d57"} Nov 02 21:59:07.625851 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.625685 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" event={"ID":"6022facf-3f59-4c38-bcb1-d7a8e3853e68","Type":"ContainerStarted","Data":"c7129b9abbf2a129adf14a2b1b10c84689bbf94a1417752a6c7445a09eaf9cb3"} Nov 02 21:59:07.632901 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.632886 4309 scope.go:122] "RemoveContainer" containerID="dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" Nov 02 21:59:07.639459 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.639443 4309 scope.go:122] "RemoveContainer" containerID="af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" Nov 02 21:59:07.646093 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.646075 4309 scope.go:122] "RemoveContainer" containerID="013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" Nov 02 21:59:07.647613 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.647591 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:59:07.651951 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.651932 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:59:07.653826 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.653812 4309 scope.go:122] "RemoveContainer" containerID="93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" Nov 02 21:59:07.662998 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.662982 4309 scope.go:122] "RemoveContainer" containerID="72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" Nov 02 21:59:07.670027 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.670010 4309 scope.go:122] "RemoveContainer" containerID="7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0" Nov 02 21:59:07.674962 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.674940 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:59:07.675279 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675264 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="init-config-reloader" Nov 02 21:59:07.675348 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675282 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="init-config-reloader" Nov 02 21:59:07.675348 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675299 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy" Nov 02 21:59:07.675348 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675306 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy" Nov 02 21:59:07.675348 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675327 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy-thanos" Nov 02 21:59:07.675348 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675335 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy-thanos" Nov 02 21:59:07.675348 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675348 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="prometheus" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675356 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="prometheus" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675367 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="config-reloader" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675375 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="config-reloader" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675386 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="thanos-sidecar" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675393 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="thanos-sidecar" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675405 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy-web" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675412 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy-web" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675485 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675497 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy-thanos" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675507 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="prometheus" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675517 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="config-reloader" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675525 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="thanos-sidecar" Nov 02 21:59:07.675658 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.675535 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" containerName="kube-rbac-proxy-web" Nov 02 21:59:07.676929 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.676910 4309 scope.go:122] "RemoveContainer" containerID="38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" Nov 02 21:59:07.677199 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:07.677181 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": container with ID starting with 38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3 not found: ID does not exist" containerID="38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" Nov 02 21:59:07.677255 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.677207 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3"} err="failed to get container status \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": rpc error: code = NotFound desc = could not find container \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": container with ID starting with 38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3 not found: ID does not exist" Nov 02 21:59:07.677255 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.677222 4309 scope.go:122] "RemoveContainer" containerID="dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" Nov 02 21:59:07.677493 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:07.677478 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": container with ID starting with dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a not found: ID does not exist" containerID="dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" Nov 02 21:59:07.677545 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.677496 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a"} err="failed to get container status \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": rpc error: code = NotFound desc = could not find container \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": container with ID starting with dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a not found: ID does not exist" Nov 02 21:59:07.677545 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.677509 4309 scope.go:122] "RemoveContainer" containerID="af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" Nov 02 21:59:07.677734 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:07.677716 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": container with ID starting with af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6 not found: ID does not exist" containerID="af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" Nov 02 21:59:07.677791 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.677737 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6"} err="failed to get container status \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": rpc error: code = NotFound desc = could not find container \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": container with ID starting with af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6 not found: ID does not exist" Nov 02 21:59:07.677791 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.677753 4309 scope.go:122] "RemoveContainer" containerID="013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" Nov 02 21:59:07.677993 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:07.677975 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": container with ID starting with 013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3 not found: ID does not exist" containerID="013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" Nov 02 21:59:07.678037 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.677996 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3"} err="failed to get container status \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": rpc error: code = NotFound desc = could not find container \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": container with ID starting with 013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3 not found: ID does not exist" Nov 02 21:59:07.678037 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.678008 4309 scope.go:122] "RemoveContainer" containerID="93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" Nov 02 21:59:07.678182 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:07.678165 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": container with ID starting with 93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d not found: ID does not exist" containerID="93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" Nov 02 21:59:07.678216 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.678186 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d"} err="failed to get container status \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": rpc error: code = NotFound desc = could not find container \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": container with ID starting with 93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d not found: ID does not exist" Nov 02 21:59:07.678216 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.678198 4309 scope.go:122] "RemoveContainer" containerID="72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" Nov 02 21:59:07.678363 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:07.678349 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": container with ID starting with 72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea not found: ID does not exist" containerID="72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" Nov 02 21:59:07.678400 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.678366 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea"} err="failed to get container status \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": rpc error: code = NotFound desc = could not find container \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": container with ID starting with 72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea not found: ID does not exist" Nov 02 21:59:07.678400 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.678376 4309 scope.go:122] "RemoveContainer" containerID="7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0" Nov 02 21:59:07.678598 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:07.678581 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": container with ID starting with 7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0 not found: ID does not exist" containerID="7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0" Nov 02 21:59:07.678643 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.678609 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0"} err="failed to get container status \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": rpc error: code = NotFound desc = could not find container \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": container with ID starting with 7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0 not found: ID does not exist" Nov 02 21:59:07.678643 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.678621 4309 scope.go:122] "RemoveContainer" containerID="38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" Nov 02 21:59:07.678843 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.678824 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3"} err="failed to get container status \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": rpc error: code = NotFound desc = could not find container \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": container with ID starting with 38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3 not found: ID does not exist" Nov 02 21:59:07.678889 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.678845 4309 scope.go:122] "RemoveContainer" containerID="dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" Nov 02 21:59:07.679050 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.679025 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a"} err="failed to get container status \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": rpc error: code = NotFound desc = could not find container \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": container with ID starting with dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a not found: ID does not exist" Nov 02 21:59:07.679140 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.679054 4309 scope.go:122] "RemoveContainer" containerID="af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" Nov 02 21:59:07.679297 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.679272 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6"} err="failed to get container status \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": rpc error: code = NotFound desc = could not find container \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": container with ID starting with af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6 not found: ID does not exist" Nov 02 21:59:07.679336 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.679299 4309 scope.go:122] "RemoveContainer" containerID="013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" Nov 02 21:59:07.679595 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.679573 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3"} err="failed to get container status \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": rpc error: code = NotFound desc = could not find container \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": container with ID starting with 013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3 not found: ID does not exist" Nov 02 21:59:07.679595 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.679593 4309 scope.go:122] "RemoveContainer" containerID="93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" Nov 02 21:59:07.679901 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.679881 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d"} err="failed to get container status \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": rpc error: code = NotFound desc = could not find container \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": container with ID starting with 93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d not found: ID does not exist" Nov 02 21:59:07.679956 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.679902 4309 scope.go:122] "RemoveContainer" containerID="72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" Nov 02 21:59:07.680124 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.680107 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea"} err="failed to get container status \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": rpc error: code = NotFound desc = could not find container \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": container with ID starting with 72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea not found: ID does not exist" Nov 02 21:59:07.680164 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.680125 4309 scope.go:122] "RemoveContainer" containerID="7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0" Nov 02 21:59:07.680364 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.680345 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0"} err="failed to get container status \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": rpc error: code = NotFound desc = could not find container \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": container with ID starting with 7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0 not found: ID does not exist" Nov 02 21:59:07.680428 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.680366 4309 scope.go:122] "RemoveContainer" containerID="38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" Nov 02 21:59:07.680627 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.680610 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3"} err="failed to get container status \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": rpc error: code = NotFound desc = could not find container \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": container with ID starting with 38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3 not found: ID does not exist" Nov 02 21:59:07.680714 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.680628 4309 scope.go:122] "RemoveContainer" containerID="dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" Nov 02 21:59:07.680880 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.680859 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a"} err="failed to get container status \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": rpc error: code = NotFound desc = could not find container \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": container with ID starting with dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a not found: ID does not exist" Nov 02 21:59:07.680923 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.680882 4309 scope.go:122] "RemoveContainer" containerID="af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" Nov 02 21:59:07.681115 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.681098 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6"} err="failed to get container status \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": rpc error: code = NotFound desc = could not find container \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": container with ID starting with af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6 not found: ID does not exist" Nov 02 21:59:07.681178 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.681116 4309 scope.go:122] "RemoveContainer" containerID="013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" Nov 02 21:59:07.681374 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.681354 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3"} err="failed to get container status \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": rpc error: code = NotFound desc = could not find container \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": container with ID starting with 013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3 not found: ID does not exist" Nov 02 21:59:07.681430 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.681374 4309 scope.go:122] "RemoveContainer" containerID="93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" Nov 02 21:59:07.681614 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.681592 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d"} err="failed to get container status \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": rpc error: code = NotFound desc = could not find container \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": container with ID starting with 93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d not found: ID does not exist" Nov 02 21:59:07.681614 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.681613 4309 scope.go:122] "RemoveContainer" containerID="72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" Nov 02 21:59:07.681857 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.681842 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea"} err="failed to get container status \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": rpc error: code = NotFound desc = could not find container \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": container with ID starting with 72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea not found: ID does not exist" Nov 02 21:59:07.681857 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.681857 4309 scope.go:122] "RemoveContainer" containerID="7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0" Nov 02 21:59:07.682064 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682048 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0"} err="failed to get container status \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": rpc error: code = NotFound desc = could not find container \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": container with ID starting with 7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0 not found: ID does not exist" Nov 02 21:59:07.682101 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682063 4309 scope.go:122] "RemoveContainer" containerID="38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" Nov 02 21:59:07.682256 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682241 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3"} err="failed to get container status \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": rpc error: code = NotFound desc = could not find container \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": container with ID starting with 38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3 not found: ID does not exist" Nov 02 21:59:07.682330 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682255 4309 scope.go:122] "RemoveContainer" containerID="dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" Nov 02 21:59:07.682433 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682418 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a"} err="failed to get container status \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": rpc error: code = NotFound desc = could not find container \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": container with ID starting with dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a not found: ID does not exist" Nov 02 21:59:07.682469 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682434 4309 scope.go:122] "RemoveContainer" containerID="af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" Nov 02 21:59:07.682602 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682587 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6"} err="failed to get container status \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": rpc error: code = NotFound desc = could not find container \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": container with ID starting with af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6 not found: ID does not exist" Nov 02 21:59:07.682638 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682601 4309 scope.go:122] "RemoveContainer" containerID="013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" Nov 02 21:59:07.682786 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682772 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3"} err="failed to get container status \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": rpc error: code = NotFound desc = could not find container \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": container with ID starting with 013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3 not found: ID does not exist" Nov 02 21:59:07.682832 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682786 4309 scope.go:122] "RemoveContainer" containerID="93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" Nov 02 21:59:07.683014 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.682999 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d"} err="failed to get container status \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": rpc error: code = NotFound desc = could not find container \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": container with ID starting with 93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d not found: ID does not exist" Nov 02 21:59:07.683050 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.683013 4309 scope.go:122] "RemoveContainer" containerID="72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" Nov 02 21:59:07.683219 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.683205 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea"} err="failed to get container status \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": rpc error: code = NotFound desc = could not find container \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": container with ID starting with 72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea not found: ID does not exist" Nov 02 21:59:07.683255 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.683219 4309 scope.go:122] "RemoveContainer" containerID="7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0" Nov 02 21:59:07.683403 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.683387 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0"} err="failed to get container status \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": rpc error: code = NotFound desc = could not find container \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": container with ID starting with 7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0 not found: ID does not exist" Nov 02 21:59:07.683442 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.683403 4309 scope.go:122] "RemoveContainer" containerID="38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" Nov 02 21:59:07.683582 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.683566 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3"} err="failed to get container status \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": rpc error: code = NotFound desc = could not find container \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": container with ID starting with 38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3 not found: ID does not exist" Nov 02 21:59:07.683582 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.683581 4309 scope.go:122] "RemoveContainer" containerID="dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" Nov 02 21:59:07.683845 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.683830 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a"} err="failed to get container status \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": rpc error: code = NotFound desc = could not find container \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": container with ID starting with dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a not found: ID does not exist" Nov 02 21:59:07.683845 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.683844 4309 scope.go:122] "RemoveContainer" containerID="af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" Nov 02 21:59:07.684035 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684017 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6"} err="failed to get container status \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": rpc error: code = NotFound desc = could not find container \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": container with ID starting with af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6 not found: ID does not exist" Nov 02 21:59:07.684073 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684037 4309 scope.go:122] "RemoveContainer" containerID="013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" Nov 02 21:59:07.684245 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684230 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3"} err="failed to get container status \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": rpc error: code = NotFound desc = could not find container \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": container with ID starting with 013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3 not found: ID does not exist" Nov 02 21:59:07.684283 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684245 4309 scope.go:122] "RemoveContainer" containerID="93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" Nov 02 21:59:07.684452 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684428 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d"} err="failed to get container status \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": rpc error: code = NotFound desc = could not find container \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": container with ID starting with 93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d not found: ID does not exist" Nov 02 21:59:07.684452 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684446 4309 scope.go:122] "RemoveContainer" containerID="72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" Nov 02 21:59:07.684747 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684719 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea"} err="failed to get container status \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": rpc error: code = NotFound desc = could not find container \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": container with ID starting with 72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea not found: ID does not exist" Nov 02 21:59:07.684747 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684734 4309 scope.go:122] "RemoveContainer" containerID="7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0" Nov 02 21:59:07.684965 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684950 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0"} err="failed to get container status \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": rpc error: code = NotFound desc = could not find container \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": container with ID starting with 7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0 not found: ID does not exist" Nov 02 21:59:07.685005 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.684965 4309 scope.go:122] "RemoveContainer" containerID="38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3" Nov 02 21:59:07.685146 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.685131 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3"} err="failed to get container status \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": rpc error: code = NotFound desc = could not find container \"38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3\": container with ID starting with 38aa13d9d75e552c2a7f8314142f2b2fa20799c565ce2f7655efea1dd66e30e3 not found: ID does not exist" Nov 02 21:59:07.685193 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.685146 4309 scope.go:122] "RemoveContainer" containerID="dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a" Nov 02 21:59:07.685363 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.685347 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a"} err="failed to get container status \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": rpc error: code = NotFound desc = could not find container \"dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a\": container with ID starting with dc1104dc008820e7cffd2742dc03759c3f90a2ec6f78457f851e74931c95cf4a not found: ID does not exist" Nov 02 21:59:07.685397 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.685364 4309 scope.go:122] "RemoveContainer" containerID="af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6" Nov 02 21:59:07.685575 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.685558 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6"} err="failed to get container status \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": rpc error: code = NotFound desc = could not find container \"af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6\": container with ID starting with af2897614bdac6d2620b3b1aedc872f05a132f5c0a786d758ecfdfadc2987fd6 not found: ID does not exist" Nov 02 21:59:07.685623 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.685576 4309 scope.go:122] "RemoveContainer" containerID="013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3" Nov 02 21:59:07.685827 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.685806 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3"} err="failed to get container status \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": rpc error: code = NotFound desc = could not find container \"013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3\": container with ID starting with 013832b3c598513deeb5a7a9c14f0d2d8bcc19ef9f0f81c1e2eb940f1788e7d3 not found: ID does not exist" Nov 02 21:59:07.685886 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.685826 4309 scope.go:122] "RemoveContainer" containerID="93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d" Nov 02 21:59:07.686092 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.686076 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d"} err="failed to get container status \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": rpc error: code = NotFound desc = could not find container \"93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d\": container with ID starting with 93abe10fb1c8d47a79e8f9df660cbdaf5096e17ff8314460f8accdc9b05d496d not found: ID does not exist" Nov 02 21:59:07.686141 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.686092 4309 scope.go:122] "RemoveContainer" containerID="72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea" Nov 02 21:59:07.686314 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.686299 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea"} err="failed to get container status \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": rpc error: code = NotFound desc = could not find container \"72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea\": container with ID starting with 72d117df25153765f532260185f970d6b810920b3edefed27863013a7c9b62ea not found: ID does not exist" Nov 02 21:59:07.686314 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.686313 4309 scope.go:122] "RemoveContainer" containerID="7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0" Nov 02 21:59:07.686522 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.686507 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0"} err="failed to get container status \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": rpc error: code = NotFound desc = could not find container \"7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0\": container with ID starting with 7fec85259195954e9ca1c532019f3cae650e249acbff4e28fccc2c5f90d6b8c0 not found: ID does not exist" Nov 02 21:59:07.712999 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.712981 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:59:07.713104 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.713094 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.719612 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.716771 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 21:59:07.719612 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.717084 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 21:59:07.719612 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.717151 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 21:59:07.719612 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.717220 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 21:59:07.719612 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.717250 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ein12ipsqq4i\"" Nov 02 21:59:07.719612 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.717732 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 21:59:07.719612 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.717736 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-552fk\"" Nov 02 21:59:07.719612 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.718063 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 21:59:07.719612 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.719033 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 21:59:07.720073 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.719738 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 21:59:07.720800 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.720771 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 21:59:07.720800 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.720770 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 21:59:07.721348 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.721325 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 21:59:07.721348 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.721328 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 21:59:07.724933 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.724912 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 21:59:07.725713 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.725697 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 21:59:07.813691 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.813640 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.813710 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b29d959-15eb-45fc-b165-cebbdc693076-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.813739 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.813760 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.813805 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.813896 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.813955 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.813978 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814003 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814023 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b29d959-15eb-45fc-b165-cebbdc693076-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814065 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814040 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814518 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814084 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814518 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814115 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814518 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814167 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-config\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814518 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814186 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814518 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814254 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814518 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814283 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85759\" (UniqueName: \"kubernetes.io/projected/3b29d959-15eb-45fc-b165-cebbdc693076-kube-api-access-85759\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814518 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814346 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814518 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814376 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.814518 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.814406 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915147 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915065 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915549 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915530 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915648 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915562 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915648 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915603 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915774 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915709 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b29d959-15eb-45fc-b165-cebbdc693076-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915774 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915756 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915876 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915782 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915876 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915807 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915876 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915844 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-config\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.915876 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915867 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916049 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915918 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916049 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915964 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85759\" (UniqueName: \"kubernetes.io/projected/3b29d959-15eb-45fc-b165-cebbdc693076-kube-api-access-85759\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916049 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.915992 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916049 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916014 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916049 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916042 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916208 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916074 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916208 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916110 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b29d959-15eb-45fc-b165-cebbdc693076-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916208 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916137 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916208 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916160 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916208 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916185 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916447 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916291 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916447 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916358 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916546 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916531 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.916806 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.916786 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.917570 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.917256 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.918917 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.918248 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.918917 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.918569 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.918917 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.918735 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b29d959-15eb-45fc-b165-cebbdc693076-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.918917 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.918839 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.919262 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.919096 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.919601 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.919538 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.920204 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.920176 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.920417 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.920393 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.921141 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.921107 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-config\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.921141 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.921119 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.922313 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.922292 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.922486 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.922470 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b29d959-15eb-45fc-b165-cebbdc693076-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.923116 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.923092 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b29d959-15eb-45fc-b165-cebbdc693076-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.924328 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.924306 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b29d959-15eb-45fc-b165-cebbdc693076-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:07.926671 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:07.926642 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85759\" (UniqueName: \"kubernetes.io/projected/3b29d959-15eb-45fc-b165-cebbdc693076-kube-api-access-85759\") pod \"prometheus-k8s-0\" (UID: \"3b29d959-15eb-45fc-b165-cebbdc693076\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:08.159185 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:08.159132 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:08.187203 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:08.187134 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a24b4fc-8ecd-4f2b-946f-a78194105d94" path="/var/lib/kubelet/pods/7a24b4fc-8ecd-4f2b-946f-a78194105d94/volumes" Nov 02 21:59:08.474943 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:08.474915 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 21:59:08.477250 ip-10-0-136-9 kubenswrapper[4309]: W1102 21:59:08.477228 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b29d959_15eb_45fc_b165_cebbdc693076.slice/crio-360fdf2f6312b7a630bf4ec566b7ef9325b12f4a126cb13c464d88a10d11794d WatchSource:0}: Error finding container 360fdf2f6312b7a630bf4ec566b7ef9325b12f4a126cb13c464d88a10d11794d: Status 404 returned error can't find the container with id 360fdf2f6312b7a630bf4ec566b7ef9325b12f4a126cb13c464d88a10d11794d Nov 02 21:59:08.633219 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:08.633190 4309 generic.go:356] "Generic (PLEG): container finished" podID="3b29d959-15eb-45fc-b165-cebbdc693076" containerID="d74e8127675e75be5b89e014f70dc6b6327a010dee4142981fcad34d264b34dc" exitCode=0 Nov 02 21:59:08.633341 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:08.633236 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b29d959-15eb-45fc-b165-cebbdc693076","Type":"ContainerDied","Data":"d74e8127675e75be5b89e014f70dc6b6327a010dee4142981fcad34d264b34dc"} Nov 02 21:59:08.633341 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:08.633306 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b29d959-15eb-45fc-b165-cebbdc693076","Type":"ContainerStarted","Data":"360fdf2f6312b7a630bf4ec566b7ef9325b12f4a126cb13c464d88a10d11794d"} Nov 02 21:59:09.640419 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.640384 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b29d959-15eb-45fc-b165-cebbdc693076","Type":"ContainerStarted","Data":"1f3422026fdffc74dd0f31100684b69de497a12c6f832e17cba5673e7330dd0f"} Nov 02 21:59:09.640888 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.640506 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b29d959-15eb-45fc-b165-cebbdc693076","Type":"ContainerStarted","Data":"577c896155f99490a4df7614cc79179c0b77cad7c3d9080b9c2e0f501e32f266"} Nov 02 21:59:09.640888 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.640554 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b29d959-15eb-45fc-b165-cebbdc693076","Type":"ContainerStarted","Data":"77740852088e7aaa043f1fe314fc2178e73a67ad6062f764b94ac7c2eef9db76"} Nov 02 21:59:09.640888 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.640594 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b29d959-15eb-45fc-b165-cebbdc693076","Type":"ContainerStarted","Data":"322526a431e5d887f9e27849a5bfc7ca5bfd88a62fd6ce191908c82f8b3a610b"} Nov 02 21:59:09.640888 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.640629 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b29d959-15eb-45fc-b165-cebbdc693076","Type":"ContainerStarted","Data":"c60db771856e40e2a788a1b524186291617d4d07ad47d24065a060853dfa8ed1"} Nov 02 21:59:09.640888 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.640663 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b29d959-15eb-45fc-b165-cebbdc693076","Type":"ContainerStarted","Data":"6498c3205c3e994ffabe096f8edd6f29b0782c0bf101493a9375bb22522e49ca"} Nov 02 21:59:09.642694 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.642650 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" event={"ID":"6022facf-3f59-4c38-bcb1-d7a8e3853e68","Type":"ContainerStarted","Data":"019fe01c04d0d6831b20d339a1e6a79ff6064eb69511ba76d3fa500fff4007d6"} Nov 02 21:59:09.642803 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.642706 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" event={"ID":"6022facf-3f59-4c38-bcb1-d7a8e3853e68","Type":"ContainerStarted","Data":"78a3baeb0fe9470969fc0c10089f59dab2b25547ced3a5bba19b4ce250fe693d"} Nov 02 21:59:09.642803 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.642724 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" event={"ID":"6022facf-3f59-4c38-bcb1-d7a8e3853e68","Type":"ContainerStarted","Data":"05080df6327287be31afe6f75f2dbfa853dc5aae5269d81a34032644cd465513"} Nov 02 21:59:09.667028 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.666978 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.652066409 podStartE2EDuration="2.666962149s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.89574ms" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:09.664566725 +0000 UTC m=+77.602679944" watchObservedRunningTime="2026-11-02 21:59:09.666962149 +0000 UTC m=+77.605075369" Nov 02 21:59:09.685090 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:09.685044 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7dc44596d8-r9jct" podStartSLOduration=1.26266799 podStartE2EDuration="3.685030969s" totalImagesPullingTime="2.422362979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:59:09.683548526 +0000 UTC m=+77.621661760" watchObservedRunningTime="2026-11-02 21:59:09.685030969 +0000 UTC m=+77.623144188" Nov 02 21:59:11.594168 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:11.594125 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 21:59:13.159664 ip-10-0-136-9 kubenswrapper[4309]: I1102 21:59:13.159632 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 21:59:41.593697 ip-10-0-136-9 kubenswrapper[4309]: E1102 21:59:41.593643 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:00:08.159749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:00:08.159628 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:00:08.175538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:00:08.175510 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:00:08.885157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:00:08.885128 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 22:00:11.594096 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:00:11.594055 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:00:41.594031 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:00:41.593992 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:01:11.592827 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:01:11.592788 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:01:41.593366 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:01:41.593284 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:02:11.593623 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:02:11.593584 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:02:41.593506 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:02:41.593466 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:02:52.192630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:02:52.192602 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:02:52.193143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:02:52.192730 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:03:09.479897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.479808 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w"] Nov 02 22:03:09.484066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.484037 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.490243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.490220 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 22:03:09.490373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.490229 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 22:03:09.490373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.490283 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 22:03:09.490373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.490296 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 22:03:09.490373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.490313 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8qjb7\"" Nov 02 22:03:09.490373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.490369 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 22:03:09.511608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.511574 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w"] Nov 02 22:03:09.624043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.623997 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/981b466e-ef40-47ae-89c9-609b5733a75d-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.624043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.624047 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/981b466e-ef40-47ae-89c9-609b5733a75d-tmp\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.624290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.624076 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/981b466e-ef40-47ae-89c9-609b5733a75d-manifests\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.624290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.624158 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/981b466e-ef40-47ae-89c9-609b5733a75d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.624290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.624206 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88slw\" (UniqueName: \"kubernetes.io/projected/981b466e-ef40-47ae-89c9-609b5733a75d-kube-api-access-88slw\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.624393 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.624305 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/981b466e-ef40-47ae-89c9-609b5733a75d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.725617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.725570 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/981b466e-ef40-47ae-89c9-609b5733a75d-tmp\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.725808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.725638 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/981b466e-ef40-47ae-89c9-609b5733a75d-manifests\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.725808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.725670 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/981b466e-ef40-47ae-89c9-609b5733a75d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.725808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.725741 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88slw\" (UniqueName: \"kubernetes.io/projected/981b466e-ef40-47ae-89c9-609b5733a75d-kube-api-access-88slw\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.725808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.725805 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/981b466e-ef40-47ae-89c9-609b5733a75d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.726015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.725849 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/981b466e-ef40-47ae-89c9-609b5733a75d-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.726174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.726146 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/981b466e-ef40-47ae-89c9-609b5733a75d-manifests\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.727942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.727920 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/981b466e-ef40-47ae-89c9-609b5733a75d-tmp\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.728270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.728250 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/981b466e-ef40-47ae-89c9-609b5733a75d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.728342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.728298 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/981b466e-ef40-47ae-89c9-609b5733a75d-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.728383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.728343 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/981b466e-ef40-47ae-89c9-609b5733a75d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.733731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.733663 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88slw\" (UniqueName: \"kubernetes.io/projected/981b466e-ef40-47ae-89c9-609b5733a75d-kube-api-access-88slw\") pod \"opendatahub-operator-controller-manager-5c6f6f47c9-6n66w\" (UID: \"981b466e-ef40-47ae-89c9-609b5733a75d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:09.798654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:09.798620 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:10.092244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:10.092166 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w"] Nov 02 22:03:10.095055 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:03:10.095013 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981b466e_ef40_47ae_89c9_609b5733a75d.slice/crio-33f80ef33ab4cc409121784ca267ac0cda6551f0711a1b870113e8340eaa2f58 WatchSource:0}: Error finding container 33f80ef33ab4cc409121784ca267ac0cda6551f0711a1b870113e8340eaa2f58: Status 404 returned error can't find the container with id 33f80ef33ab4cc409121784ca267ac0cda6551f0711a1b870113e8340eaa2f58 Nov 02 22:03:10.096969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:10.096948 4309 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:03:10.531629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:10.531544 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" event={"ID":"981b466e-ef40-47ae-89c9-609b5733a75d","Type":"ContainerStarted","Data":"33f80ef33ab4cc409121784ca267ac0cda6551f0711a1b870113e8340eaa2f58"} Nov 02 22:03:11.596084 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:03:11.596048 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:03:17.567349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:17.567313 4309 generic.go:356] "Generic (PLEG): container finished" podID="981b466e-ef40-47ae-89c9-609b5733a75d" containerID="c5da6b88592bd5188243ba9be47931f80a86fb516e796580a404904f7bf69617" exitCode=0 Nov 02 22:03:17.567349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:17.567351 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" event={"ID":"981b466e-ef40-47ae-89c9-609b5733a75d","Type":"ContainerDied","Data":"c5da6b88592bd5188243ba9be47931f80a86fb516e796580a404904f7bf69617"} Nov 02 22:03:18.573228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:18.573189 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" event={"ID":"981b466e-ef40-47ae-89c9-609b5733a75d","Type":"ContainerStarted","Data":"afad64ac43199ca451baa556905389c7789b99d08e124bbc21cce3fd75516330"} Nov 02 22:03:18.573725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:18.573301 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:18.606522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:18.606466 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" podStartSLOduration=1.882050085 podStartE2EDuration="9.606452138s" totalImagesPullingTime="7.233042345s" totalInitContainerRuntime="491.359708ms" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:18.604144219 +0000 UTC m=+326.542257471" watchObservedRunningTime="2026-11-02 22:03:18.606452138 +0000 UTC m=+326.544565410" Nov 02 22:03:29.579329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:29.579300 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c6f6f47c9-6n66w" Nov 02 22:03:31.030039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.030003 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f"] Nov 02 22:03:31.036768 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.036743 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" Nov 02 22:03:31.043250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.043223 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-vqlmv\"" Nov 02 22:03:31.043406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.043247 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 22:03:31.043406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.043262 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 22:03:31.044880 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.044860 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f"] Nov 02 22:03:31.126578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.126541 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2p7\" (UniqueName: \"kubernetes.io/projected/315cd700-559d-4511-812c-c55c7d7a2bb8-kube-api-access-xd2p7\") pod \"jobset-operator-674cd59dc5-l5l2f\" (UID: \"315cd700-559d-4511-812c-c55c7d7a2bb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" Nov 02 22:03:31.126780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.126614 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/315cd700-559d-4511-812c-c55c7d7a2bb8-tmp\") pod \"jobset-operator-674cd59dc5-l5l2f\" (UID: \"315cd700-559d-4511-812c-c55c7d7a2bb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" Nov 02 22:03:31.227606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.227567 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2p7\" (UniqueName: \"kubernetes.io/projected/315cd700-559d-4511-812c-c55c7d7a2bb8-kube-api-access-xd2p7\") pod \"jobset-operator-674cd59dc5-l5l2f\" (UID: \"315cd700-559d-4511-812c-c55c7d7a2bb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" Nov 02 22:03:31.227776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.227649 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/315cd700-559d-4511-812c-c55c7d7a2bb8-tmp\") pod \"jobset-operator-674cd59dc5-l5l2f\" (UID: \"315cd700-559d-4511-812c-c55c7d7a2bb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" Nov 02 22:03:31.228011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.227994 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/315cd700-559d-4511-812c-c55c7d7a2bb8-tmp\") pod \"jobset-operator-674cd59dc5-l5l2f\" (UID: \"315cd700-559d-4511-812c-c55c7d7a2bb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" Nov 02 22:03:31.248537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.248498 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2p7\" (UniqueName: \"kubernetes.io/projected/315cd700-559d-4511-812c-c55c7d7a2bb8-kube-api-access-xd2p7\") pod \"jobset-operator-674cd59dc5-l5l2f\" (UID: \"315cd700-559d-4511-812c-c55c7d7a2bb8\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" Nov 02 22:03:31.362186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.362152 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" Nov 02 22:03:31.630860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:31.630788 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f"] Nov 02 22:03:31.633988 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:03:31.633961 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod315cd700_559d_4511_812c_c55c7d7a2bb8.slice/crio-a55aa14ac98656679074ef2309d3be5269965a4909ced11647ebcbaecf6ba28c WatchSource:0}: Error finding container a55aa14ac98656679074ef2309d3be5269965a4909ced11647ebcbaecf6ba28c: Status 404 returned error can't find the container with id a55aa14ac98656679074ef2309d3be5269965a4909ced11647ebcbaecf6ba28c Nov 02 22:03:32.629526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:32.629493 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" event={"ID":"315cd700-559d-4511-812c-c55c7d7a2bb8","Type":"ContainerStarted","Data":"a55aa14ac98656679074ef2309d3be5269965a4909ced11647ebcbaecf6ba28c"} Nov 02 22:03:33.359099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.359065 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-585h6"] Nov 02 22:03:33.362578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.362557 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:33.371293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.369737 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 22:03:33.371293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.369757 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-htjjv\"" Nov 02 22:03:33.371293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.369802 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 22:03:33.371293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.371246 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-585h6"] Nov 02 22:03:33.448995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.448961 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3768a351-a0c8-4472-8769-d39331d91f5a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-585h6\" (UID: \"3768a351-a0c8-4472-8769-d39331d91f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:33.448995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.448999 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5gh\" (UniqueName: \"kubernetes.io/projected/3768a351-a0c8-4472-8769-d39331d91f5a-kube-api-access-jq5gh\") pod \"cert-manager-webhook-5dbbbf75cc-585h6\" (UID: \"3768a351-a0c8-4472-8769-d39331d91f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:33.545374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.545341 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk"] Nov 02 22:03:33.549027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.548999 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" Nov 02 22:03:33.549559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.549535 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3768a351-a0c8-4472-8769-d39331d91f5a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-585h6\" (UID: \"3768a351-a0c8-4472-8769-d39331d91f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:33.549665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.549570 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5gh\" (UniqueName: \"kubernetes.io/projected/3768a351-a0c8-4472-8769-d39331d91f5a-kube-api-access-jq5gh\") pod \"cert-manager-webhook-5dbbbf75cc-585h6\" (UID: \"3768a351-a0c8-4472-8769-d39331d91f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:33.557784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.557762 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-v8kzn\"" Nov 02 22:03:33.561867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.561844 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk"] Nov 02 22:03:33.566800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.566780 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3768a351-a0c8-4472-8769-d39331d91f5a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-585h6\" (UID: \"3768a351-a0c8-4472-8769-d39331d91f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:33.567095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.567074 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5gh\" (UniqueName: \"kubernetes.io/projected/3768a351-a0c8-4472-8769-d39331d91f5a-kube-api-access-jq5gh\") pod \"cert-manager-webhook-5dbbbf75cc-585h6\" (UID: \"3768a351-a0c8-4472-8769-d39331d91f5a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:33.655614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.655537 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99104650-3261-4b82-9022-2c3720b47f4f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-srtmk\" (UID: \"99104650-3261-4b82-9022-2c3720b47f4f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" Nov 02 22:03:33.655614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.655571 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qls2d\" (UniqueName: \"kubernetes.io/projected/99104650-3261-4b82-9022-2c3720b47f4f-kube-api-access-qls2d\") pod \"cert-manager-cainjector-5c57bbd4fc-srtmk\" (UID: \"99104650-3261-4b82-9022-2c3720b47f4f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" Nov 02 22:03:33.677224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.677201 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:33.756318 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.756284 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99104650-3261-4b82-9022-2c3720b47f4f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-srtmk\" (UID: \"99104650-3261-4b82-9022-2c3720b47f4f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" Nov 02 22:03:33.756318 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.756320 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qls2d\" (UniqueName: \"kubernetes.io/projected/99104650-3261-4b82-9022-2c3720b47f4f-kube-api-access-qls2d\") pod \"cert-manager-cainjector-5c57bbd4fc-srtmk\" (UID: \"99104650-3261-4b82-9022-2c3720b47f4f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" Nov 02 22:03:33.767969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.767941 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qls2d\" (UniqueName: \"kubernetes.io/projected/99104650-3261-4b82-9022-2c3720b47f4f-kube-api-access-qls2d\") pod \"cert-manager-cainjector-5c57bbd4fc-srtmk\" (UID: \"99104650-3261-4b82-9022-2c3720b47f4f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" Nov 02 22:03:33.768500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.768478 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/99104650-3261-4b82-9022-2c3720b47f4f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-srtmk\" (UID: \"99104650-3261-4b82-9022-2c3720b47f4f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" Nov 02 22:03:33.863061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.863024 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" Nov 02 22:03:33.960779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:33.960750 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-585h6"] Nov 02 22:03:33.963189 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:03:33.963151 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3768a351_a0c8_4472_8769_d39331d91f5a.slice/crio-1565b93e61f79204d0b71fb9ff6eb0e75e66b6268b995057ed87c0e6ad27bff2 WatchSource:0}: Error finding container 1565b93e61f79204d0b71fb9ff6eb0e75e66b6268b995057ed87c0e6ad27bff2: Status 404 returned error can't find the container with id 1565b93e61f79204d0b71fb9ff6eb0e75e66b6268b995057ed87c0e6ad27bff2 Nov 02 22:03:34.152090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:34.152064 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk"] Nov 02 22:03:34.154821 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:03:34.154794 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99104650_3261_4b82_9022_2c3720b47f4f.slice/crio-47d6a2afaa0c49df81ce56721f1a086b0fcc4aa379da380ce939916639c5520b WatchSource:0}: Error finding container 47d6a2afaa0c49df81ce56721f1a086b0fcc4aa379da380ce939916639c5520b: Status 404 returned error can't find the container with id 47d6a2afaa0c49df81ce56721f1a086b0fcc4aa379da380ce939916639c5520b Nov 02 22:03:34.639275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:34.639237 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" event={"ID":"3768a351-a0c8-4472-8769-d39331d91f5a","Type":"ContainerStarted","Data":"1565b93e61f79204d0b71fb9ff6eb0e75e66b6268b995057ed87c0e6ad27bff2"} Nov 02 22:03:34.640391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:34.640370 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" event={"ID":"99104650-3261-4b82-9022-2c3720b47f4f","Type":"ContainerStarted","Data":"47d6a2afaa0c49df81ce56721f1a086b0fcc4aa379da380ce939916639c5520b"} Nov 02 22:03:39.666001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:39.665963 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" event={"ID":"3768a351-a0c8-4472-8769-d39331d91f5a","Type":"ContainerStarted","Data":"b366ac8b53c93e029218406942e3c59df5474e8f74ca76b634f89cb75e4724e3"} Nov 02 22:03:39.666437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:39.666088 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:39.683942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:39.683885 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" podStartSLOduration=1.740184953 podStartE2EDuration="6.683870633s" totalImagesPullingTime="4.94368568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:39.681405859 +0000 UTC m=+347.619519077" watchObservedRunningTime="2026-11-02 22:03:39.683870633 +0000 UTC m=+347.621983851" Nov 02 22:03:41.604644 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:03:41.604603 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:03:42.681821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:42.681788 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" event={"ID":"315cd700-559d-4511-812c-c55c7d7a2bb8","Type":"ContainerStarted","Data":"803de802835a0b859ab9e0afbb308957fd4c8547223badeae0f1f93da20d3e0e"} Nov 02 22:03:42.683489 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:42.683465 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" event={"ID":"99104650-3261-4b82-9022-2c3720b47f4f","Type":"ContainerStarted","Data":"55a3e1880be47a1d344262d68a018b7eb581f5ca71fa4e23f8fc0f7ce5406e0b"} Nov 02 22:03:42.700042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:42.699996 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-l5l2f" podStartSLOduration=1.398746396 podStartE2EDuration="11.699984452s" totalImagesPullingTime="10.301238056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:42.698239126 +0000 UTC m=+350.636352344" watchObservedRunningTime="2026-11-02 22:03:42.699984452 +0000 UTC m=+350.638097670" Nov 02 22:03:42.715658 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:42.715614 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-srtmk" podStartSLOduration=2.040063114 podStartE2EDuration="9.715603127s" totalImagesPullingTime="7.675540013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:42.714181082 +0000 UTC m=+350.652294301" watchObservedRunningTime="2026-11-02 22:03:42.715603127 +0000 UTC m=+350.653716345" Nov 02 22:03:45.672826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:45.672790 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" Nov 02 22:03:50.661179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.661147 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq"] Nov 02 22:03:50.669487 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.669463 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.689296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.689270 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 22:03:50.689296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.689283 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 22:03:50.689446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.689347 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 22:03:50.689446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.689351 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 22:03:50.689446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.689396 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-m49qh\"" Nov 02 22:03:50.692475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.692452 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq"] Nov 02 22:03:50.815816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.815775 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5nqv\" (UniqueName: \"kubernetes.io/projected/e7d8a26d-d0db-4eb3-b5de-75a070848241-kube-api-access-z5nqv\") pod \"kube-auth-proxy-5747dfd5df-bxjlq\" (UID: \"e7d8a26d-d0db-4eb3-b5de-75a070848241\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.816003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.815922 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7d8a26d-d0db-4eb3-b5de-75a070848241-tls-certs\") pod \"kube-auth-proxy-5747dfd5df-bxjlq\" (UID: \"e7d8a26d-d0db-4eb3-b5de-75a070848241\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.816075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.816009 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7d8a26d-d0db-4eb3-b5de-75a070848241-tmp\") pod \"kube-auth-proxy-5747dfd5df-bxjlq\" (UID: \"e7d8a26d-d0db-4eb3-b5de-75a070848241\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.916571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.916485 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5nqv\" (UniqueName: \"kubernetes.io/projected/e7d8a26d-d0db-4eb3-b5de-75a070848241-kube-api-access-z5nqv\") pod \"kube-auth-proxy-5747dfd5df-bxjlq\" (UID: \"e7d8a26d-d0db-4eb3-b5de-75a070848241\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.916733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.916581 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7d8a26d-d0db-4eb3-b5de-75a070848241-tls-certs\") pod \"kube-auth-proxy-5747dfd5df-bxjlq\" (UID: \"e7d8a26d-d0db-4eb3-b5de-75a070848241\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.916733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.916611 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7d8a26d-d0db-4eb3-b5de-75a070848241-tmp\") pod \"kube-auth-proxy-5747dfd5df-bxjlq\" (UID: \"e7d8a26d-d0db-4eb3-b5de-75a070848241\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.918854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.918819 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e7d8a26d-d0db-4eb3-b5de-75a070848241-tmp\") pod \"kube-auth-proxy-5747dfd5df-bxjlq\" (UID: \"e7d8a26d-d0db-4eb3-b5de-75a070848241\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.919133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.919117 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7d8a26d-d0db-4eb3-b5de-75a070848241-tls-certs\") pod \"kube-auth-proxy-5747dfd5df-bxjlq\" (UID: \"e7d8a26d-d0db-4eb3-b5de-75a070848241\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.924358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.924327 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5nqv\" (UniqueName: \"kubernetes.io/projected/e7d8a26d-d0db-4eb3-b5de-75a070848241-kube-api-access-z5nqv\") pod \"kube-auth-proxy-5747dfd5df-bxjlq\" (UID: \"e7d8a26d-d0db-4eb3-b5de-75a070848241\") " pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:50.980823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:50.980792 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" Nov 02 22:03:51.250964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:51.250883 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq"] Nov 02 22:03:51.253899 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:03:51.253874 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d8a26d_d0db_4eb3_b5de_75a070848241.slice/crio-bd92ebb6487c51f46bb6c81a59dc44b0ac0441d72447f798da2b4542a8e09e71 WatchSource:0}: Error finding container bd92ebb6487c51f46bb6c81a59dc44b0ac0441d72447f798da2b4542a8e09e71: Status 404 returned error can't find the container with id bd92ebb6487c51f46bb6c81a59dc44b0ac0441d72447f798da2b4542a8e09e71 Nov 02 22:03:51.726637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:51.726602 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" event={"ID":"e7d8a26d-d0db-4eb3-b5de-75a070848241","Type":"ContainerStarted","Data":"bd92ebb6487c51f46bb6c81a59dc44b0ac0441d72447f798da2b4542a8e09e71"} Nov 02 22:03:53.769291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:53.769268 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 22:03:54.740611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:54.740577 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" event={"ID":"e7d8a26d-d0db-4eb3-b5de-75a070848241","Type":"ContainerStarted","Data":"0b122ecd23647511754e5a01f0b422394e2e4bb5d1029890e1cdab241f9fbfa5"} Nov 02 22:03:54.756627 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:54.756578 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5747dfd5df-bxjlq" podStartSLOduration=2.246380339 podStartE2EDuration="4.756563643s" totalImagesPullingTime="2.510183304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:03:54.755564782 +0000 UTC m=+362.693677999" watchObservedRunningTime="2026-11-02 22:03:54.756563643 +0000 UTC m=+362.694676861" Nov 02 22:03:57.517669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.517632 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh"] Nov 02 22:03:57.521250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.521230 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.529488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.529465 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-9wzmx\"" Nov 02 22:03:57.529654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.529603 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 22:03:57.529654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.529631 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 22:03:57.530392 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.530372 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh"] Nov 02 22:03:57.543190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.543169 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 22:03:57.684941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.684906 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d091e2d4-a383-49f6-b8fb-4ea544b231b5-metrics-certs\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.685139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.684964 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6bj\" (UniqueName: \"kubernetes.io/projected/d091e2d4-a383-49f6-b8fb-4ea544b231b5-kube-api-access-2n6bj\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.685139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.685101 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d091e2d4-a383-49f6-b8fb-4ea544b231b5-manager-config\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.685234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.685144 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d091e2d4-a383-49f6-b8fb-4ea544b231b5-cert\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.785722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.785605 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d091e2d4-a383-49f6-b8fb-4ea544b231b5-metrics-certs\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.785722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.785714 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6bj\" (UniqueName: \"kubernetes.io/projected/d091e2d4-a383-49f6-b8fb-4ea544b231b5-kube-api-access-2n6bj\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.785955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.785777 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d091e2d4-a383-49f6-b8fb-4ea544b231b5-manager-config\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.785955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.785810 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d091e2d4-a383-49f6-b8fb-4ea544b231b5-cert\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.786450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.786429 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d091e2d4-a383-49f6-b8fb-4ea544b231b5-manager-config\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.788076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.788052 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d091e2d4-a383-49f6-b8fb-4ea544b231b5-cert\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.788205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.788187 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d091e2d4-a383-49f6-b8fb-4ea544b231b5-metrics-certs\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.794532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.794504 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6bj\" (UniqueName: \"kubernetes.io/projected/d091e2d4-a383-49f6-b8fb-4ea544b231b5-kube-api-access-2n6bj\") pod \"jobset-controller-manager-7bd56b5fcb-jnfnh\" (UID: \"d091e2d4-a383-49f6-b8fb-4ea544b231b5\") " pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.834437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.834410 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:03:57.922636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.922606 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7"] Nov 02 22:03:57.926219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.926196 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:57.934872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.933165 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7"] Nov 02 22:03:57.934872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.934689 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 22:03:57.934872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.934804 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vjchw\"" Nov 02 22:03:57.944147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.944106 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 22:03:57.944279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.944155 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 22:03:57.944279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:57.944207 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 22:03:58.088064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.088035 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21e8f035-97b3-4a9e-b0c0-e91946d9d92d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-97ts7\" (UID: \"21e8f035-97b3-4a9e-b0c0-e91946d9d92d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.088237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.088070 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlwgm\" (UniqueName: \"kubernetes.io/projected/21e8f035-97b3-4a9e-b0c0-e91946d9d92d-kube-api-access-wlwgm\") pod \"kuadrant-console-plugin-66c4fb575d-97ts7\" (UID: \"21e8f035-97b3-4a9e-b0c0-e91946d9d92d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.088237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.088147 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21e8f035-97b3-4a9e-b0c0-e91946d9d92d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-97ts7\" (UID: \"21e8f035-97b3-4a9e-b0c0-e91946d9d92d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.104394 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.104371 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh"] Nov 02 22:03:58.107052 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:03:58.107028 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd091e2d4_a383_49f6_b8fb_4ea544b231b5.slice/crio-6ef54405f819f0ebdc3042a0a226287bdd109dbc0d2795362b508d1b4a1dab79 WatchSource:0}: Error finding container 6ef54405f819f0ebdc3042a0a226287bdd109dbc0d2795362b508d1b4a1dab79: Status 404 returned error can't find the container with id 6ef54405f819f0ebdc3042a0a226287bdd109dbc0d2795362b508d1b4a1dab79 Nov 02 22:03:58.189548 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.189522 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21e8f035-97b3-4a9e-b0c0-e91946d9d92d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-97ts7\" (UID: \"21e8f035-97b3-4a9e-b0c0-e91946d9d92d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.189718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.189556 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwgm\" (UniqueName: \"kubernetes.io/projected/21e8f035-97b3-4a9e-b0c0-e91946d9d92d-kube-api-access-wlwgm\") pod \"kuadrant-console-plugin-66c4fb575d-97ts7\" (UID: \"21e8f035-97b3-4a9e-b0c0-e91946d9d92d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.189718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.189587 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21e8f035-97b3-4a9e-b0c0-e91946d9d92d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-97ts7\" (UID: \"21e8f035-97b3-4a9e-b0c0-e91946d9d92d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.190139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.190115 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21e8f035-97b3-4a9e-b0c0-e91946d9d92d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-97ts7\" (UID: \"21e8f035-97b3-4a9e-b0c0-e91946d9d92d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.191836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.191815 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21e8f035-97b3-4a9e-b0c0-e91946d9d92d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-97ts7\" (UID: \"21e8f035-97b3-4a9e-b0c0-e91946d9d92d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.196523 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.196503 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwgm\" (UniqueName: \"kubernetes.io/projected/21e8f035-97b3-4a9e-b0c0-e91946d9d92d-kube-api-access-wlwgm\") pod \"kuadrant-console-plugin-66c4fb575d-97ts7\" (UID: \"21e8f035-97b3-4a9e-b0c0-e91946d9d92d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.239482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.239455 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" Nov 02 22:03:58.509819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.509737 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7"] Nov 02 22:03:58.513834 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:03:58.513804 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21e8f035_97b3_4a9e_b0c0_e91946d9d92d.slice/crio-1576426e9d7b9e8a9bdbd0bfa333798cb8b437a7082ba772bf8092f557a8949d WatchSource:0}: Error finding container 1576426e9d7b9e8a9bdbd0bfa333798cb8b437a7082ba772bf8092f557a8949d: Status 404 returned error can't find the container with id 1576426e9d7b9e8a9bdbd0bfa333798cb8b437a7082ba772bf8092f557a8949d Nov 02 22:03:58.758871 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.758839 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" event={"ID":"21e8f035-97b3-4a9e-b0c0-e91946d9d92d","Type":"ContainerStarted","Data":"1576426e9d7b9e8a9bdbd0bfa333798cb8b437a7082ba772bf8092f557a8949d"} Nov 02 22:03:58.760003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:03:58.759937 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" event={"ID":"d091e2d4-a383-49f6-b8fb-4ea544b231b5","Type":"ContainerStarted","Data":"6ef54405f819f0ebdc3042a0a226287bdd109dbc0d2795362b508d1b4a1dab79"} Nov 02 22:04:02.785116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:02.785077 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" event={"ID":"d091e2d4-a383-49f6-b8fb-4ea544b231b5","Type":"ContainerStarted","Data":"af8a11ed4b75176df8fa3e5d2947dd7eab1a0dab83a2d871439f403826016064"} Nov 02 22:04:02.785598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:02.785256 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:04:02.801391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:02.801338 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" podStartSLOduration=2.169250821 podStartE2EDuration="5.801321688s" totalImagesPullingTime="3.632070867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:02.800399688 +0000 UTC m=+370.738512905" watchObservedRunningTime="2026-11-02 22:04:02.801321688 +0000 UTC m=+370.739434905" Nov 02 22:04:04.795915 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:04.795879 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" event={"ID":"21e8f035-97b3-4a9e-b0c0-e91946d9d92d","Type":"ContainerStarted","Data":"9a3c749183680bd3c94a79b87528da329eed1cba7d7697bc18000478acc514fb"} Nov 02 22:04:04.811944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:04.811898 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-97ts7" podStartSLOduration=2.314030175 podStartE2EDuration="7.811882999s" totalImagesPullingTime="5.497852824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:04.811069881 +0000 UTC m=+372.749183099" watchObservedRunningTime="2026-11-02 22:04:04.811882999 +0000 UTC m=+372.749996217" Nov 02 22:04:06.939422 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:06.939392 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm"] Nov 02 22:04:06.942743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:06.942727 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:06.946499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:06.946478 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7kl6p\"" Nov 02 22:04:06.954557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:06.954530 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm"] Nov 02 22:04:06.973195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:06.973174 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls8n2\" (UniqueName: \"kubernetes.io/projected/1b529273-448c-4adb-9909-5f0e5faa5ca0-kube-api-access-ls8n2\") pod \"kuadrant-operator-controller-manager-678d87766f-4svmm\" (UID: \"1b529273-448c-4adb-9909-5f0e5faa5ca0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:06.973289 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:06.973213 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b529273-448c-4adb-9909-5f0e5faa5ca0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4svmm\" (UID: \"1b529273-448c-4adb-9909-5f0e5faa5ca0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:07.074519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:07.074486 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls8n2\" (UniqueName: \"kubernetes.io/projected/1b529273-448c-4adb-9909-5f0e5faa5ca0-kube-api-access-ls8n2\") pod \"kuadrant-operator-controller-manager-678d87766f-4svmm\" (UID: \"1b529273-448c-4adb-9909-5f0e5faa5ca0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:07.074698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:07.074531 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b529273-448c-4adb-9909-5f0e5faa5ca0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4svmm\" (UID: \"1b529273-448c-4adb-9909-5f0e5faa5ca0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:07.074982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:07.074964 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b529273-448c-4adb-9909-5f0e5faa5ca0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4svmm\" (UID: \"1b529273-448c-4adb-9909-5f0e5faa5ca0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:07.082695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:07.082649 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls8n2\" (UniqueName: \"kubernetes.io/projected/1b529273-448c-4adb-9909-5f0e5faa5ca0-kube-api-access-ls8n2\") pod \"kuadrant-operator-controller-manager-678d87766f-4svmm\" (UID: \"1b529273-448c-4adb-9909-5f0e5faa5ca0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:07.255730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:07.255640 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:07.534924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:07.534897 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm"] Nov 02 22:04:07.537577 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:04:07.537552 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b529273_448c_4adb_9909_5f0e5faa5ca0.slice/crio-2f4066d35808a9aba4f0e65b91992d3ac8887c87dbc9d07dec304be1b964015b WatchSource:0}: Error finding container 2f4066d35808a9aba4f0e65b91992d3ac8887c87dbc9d07dec304be1b964015b: Status 404 returned error can't find the container with id 2f4066d35808a9aba4f0e65b91992d3ac8887c87dbc9d07dec304be1b964015b Nov 02 22:04:07.810143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:07.810112 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" event={"ID":"1b529273-448c-4adb-9909-5f0e5faa5ca0","Type":"ContainerStarted","Data":"2f4066d35808a9aba4f0e65b91992d3ac8887c87dbc9d07dec304be1b964015b"} Nov 02 22:04:11.601277 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:04:11.601237 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:04:12.841665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:12.841616 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" event={"ID":"1b529273-448c-4adb-9909-5f0e5faa5ca0","Type":"ContainerStarted","Data":"77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042"} Nov 02 22:04:12.842092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:12.841807 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:12.862877 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:12.862827 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" podStartSLOduration=2.367621347 podStartE2EDuration="6.862812915s" totalImagesPullingTime="4.495191568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:04:12.859547639 +0000 UTC m=+380.797660857" watchObservedRunningTime="2026-11-02 22:04:12.862812915 +0000 UTC m=+380.800926133" Nov 02 22:04:13.795396 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:13.795367 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7bd56b5fcb-jnfnh" Nov 02 22:04:23.849293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:04:23.849263 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:04:41.593278 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:04:41.593195 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:05:11.593629 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:05:11.593583 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:05:41.593912 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:05:41.593874 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:06:11.593625 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:06:11.593538 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:06:41.593985 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:06:41.593950 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:06:44.303511 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.303472 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx"] Nov 02 22:06:44.306917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.306898 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.318670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.318649 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx"] Nov 02 22:06:44.318983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.318960 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bf7n2\"" Nov 02 22:06:44.319095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.318982 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 22:06:44.319095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.318997 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 22:06:44.319095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.319007 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 22:06:44.339858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.339829 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.339987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.339903 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.339987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.339931 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.340107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.340068 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.340164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.340112 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.340164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.340133 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf6n5\" (UniqueName: \"kubernetes.io/projected/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-kube-api-access-wf6n5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.441191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.441160 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.441332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.441233 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.441332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.441264 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.441418 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.441396 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.441467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.441455 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.441532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.441492 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf6n5\" (UniqueName: \"kubernetes.io/projected/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-kube-api-access-wf6n5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.441691 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.441651 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.441771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.441748 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.442137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.442117 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.443693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.443651 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.443973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.443947 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.448773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.448749 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf6n5\" (UniqueName: \"kubernetes.io/projected/d2b2a428-093b-4d9c-8b88-ad2b24b447c8-kube-api-access-wf6n5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-df7vx\" (UID: \"d2b2a428-093b-4d9c-8b88-ad2b24b447c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.621037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.621002 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:44.824866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.824835 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l"] Nov 02 22:06:44.828817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.828695 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:44.839077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.838664 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l"] Nov 02 22:06:44.839868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.839848 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 22:06:44.939919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.939897 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx"] Nov 02 22:06:44.942136 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:06:44.942107 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b2a428_093b_4d9c_8b88_ad2b24b447c8.slice/crio-f7a55eba52d2037842517f74f55f36469431f31e716d61f5625f6ff099e4a7c8 WatchSource:0}: Error finding container f7a55eba52d2037842517f74f55f36469431f31e716d61f5625f6ff099e4a7c8: Status 404 returned error can't find the container with id f7a55eba52d2037842517f74f55f36469431f31e716d61f5625f6ff099e4a7c8 Nov 02 22:06:44.945454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.945419 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:44.945568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.945457 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:44.945629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.945562 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7nr\" (UniqueName: \"kubernetes.io/projected/18e45205-8cd3-40c2-b688-6d31bc145c0a-kube-api-access-hm7nr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:44.945629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.945607 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e45205-8cd3-40c2-b688-6d31bc145c0a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:44.945750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.945709 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:44.945750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:44.945737 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.046649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.046614 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.046834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.046711 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7nr\" (UniqueName: \"kubernetes.io/projected/18e45205-8cd3-40c2-b688-6d31bc145c0a-kube-api-access-hm7nr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.046834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.046773 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e45205-8cd3-40c2-b688-6d31bc145c0a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.046834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.046824 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.047000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.046849 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.047000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.046887 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.047196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.047120 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.047245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.047223 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.047300 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.047281 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.049054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.049032 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e45205-8cd3-40c2-b688-6d31bc145c0a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.049312 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.049294 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e45205-8cd3-40c2-b688-6d31bc145c0a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.057450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.057427 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7nr\" (UniqueName: \"kubernetes.io/projected/18e45205-8cd3-40c2-b688-6d31bc145c0a-kube-api-access-hm7nr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xgv2l\" (UID: \"18e45205-8cd3-40c2-b688-6d31bc145c0a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.147905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.147872 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:45.429586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.429555 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l"] Nov 02 22:06:45.507701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.507588 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" event={"ID":"d2b2a428-093b-4d9c-8b88-ad2b24b447c8","Type":"ContainerStarted","Data":"f7a55eba52d2037842517f74f55f36469431f31e716d61f5625f6ff099e4a7c8"} Nov 02 22:06:45.509140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:45.509080 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" event={"ID":"18e45205-8cd3-40c2-b688-6d31bc145c0a","Type":"ContainerStarted","Data":"1963700f86b6ada802b75fbdbe7eb975225ae94e52910c3d19afb1fdd9486c17"} Nov 02 22:06:46.516215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:46.516177 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" event={"ID":"18e45205-8cd3-40c2-b688-6d31bc145c0a","Type":"ContainerStarted","Data":"2589a4ec25da68394cadf2eea831c688536ea479829b553702221979a945bf8f"} Nov 02 22:06:46.517964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:46.517932 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" event={"ID":"d2b2a428-093b-4d9c-8b88-ad2b24b447c8","Type":"ContainerStarted","Data":"313a0a20930a027b75e7856ee78617dbe926d0944bda9dcb41a5e11b0dd3c8f4"} Nov 02 22:06:46.534713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:46.534645 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" podStartSLOduration=1.645877365 podStartE2EDuration="2.534628535s" totalImagesPullingTime="888.75117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:46.533864484 +0000 UTC m=+534.471977701" watchObservedRunningTime="2026-11-02 22:06:46.534628535 +0000 UTC m=+534.472741756" Nov 02 22:06:46.550379 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:46.550297 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" podStartSLOduration=1.205513934 podStartE2EDuration="2.550283518s" totalImagesPullingTime="1.344769584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:46.549765719 +0000 UTC m=+534.487878940" watchObservedRunningTime="2026-11-02 22:06:46.550283518 +0000 UTC m=+534.488396735" Nov 02 22:06:46.622000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:46.621969 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:47.148528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:47.148480 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:48.829534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.829500 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hzbck"] Nov 02 22:06:48.836422 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.836398 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.843704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.843628 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 22:06:48.845008 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.844964 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hzbck"] Nov 02 22:06:48.879442 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.879404 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-home\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.879577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.879460 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.879623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.879608 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.879660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.879637 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzszw\" (UniqueName: \"kubernetes.io/projected/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-kube-api-access-tzszw\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.880105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.879737 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.880105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.879796 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.980502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.980471 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.980502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.980512 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-home\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.980743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.980534 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.980743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.980575 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.980743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.980594 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzszw\" (UniqueName: \"kubernetes.io/projected/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-kube-api-access-tzszw\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.980743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.980635 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.981012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.980989 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-home\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.981074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.980993 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.981074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.981039 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.982985 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.982955 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.983127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.983111 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:48.988536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:48.988502 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzszw\" (UniqueName: \"kubernetes.io/projected/774397e4-c8ee-4734-8f5a-7befd9b4d9dc-kube-api-access-tzszw\") pod \"echo-simulator-kserve-64b6f8f96c-hzbck\" (UID: \"774397e4-c8ee-4734-8f5a-7befd9b4d9dc\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:49.154313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:49.154231 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:49.432129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:49.432104 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hzbck"] Nov 02 22:06:49.434080 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:06:49.434043 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774397e4_c8ee_4734_8f5a_7befd9b4d9dc.slice/crio-43cb9c57ed6ad9a62cd1f5ac06e9eea049c757ada9f6b589a67f9af3151be35c WatchSource:0}: Error finding container 43cb9c57ed6ad9a62cd1f5ac06e9eea049c757ada9f6b589a67f9af3151be35c: Status 404 returned error can't find the container with id 43cb9c57ed6ad9a62cd1f5ac06e9eea049c757ada9f6b589a67f9af3151be35c Nov 02 22:06:49.534907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:49.534876 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" event={"ID":"774397e4-c8ee-4734-8f5a-7befd9b4d9dc","Type":"ContainerStarted","Data":"43cb9c57ed6ad9a62cd1f5ac06e9eea049c757ada9f6b589a67f9af3151be35c"} Nov 02 22:06:50.680016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.679983 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl"] Nov 02 22:06:50.688046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.688015 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.693386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.693316 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl"] Nov 02 22:06:50.697264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.697228 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 22:06:50.798994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.798954 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.798994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.799002 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.799233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.799070 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e4208b-1bbe-4a26-a5c0-77783d8239b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.799233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.799102 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9tmn\" (UniqueName: \"kubernetes.io/projected/96e4208b-1bbe-4a26-a5c0-77783d8239b9-kube-api-access-g9tmn\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.799233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.799163 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.799233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.799203 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.899719 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.899656 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.899912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.899726 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.899912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.899794 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e4208b-1bbe-4a26-a5c0-77783d8239b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.899912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.899823 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9tmn\" (UniqueName: \"kubernetes.io/projected/96e4208b-1bbe-4a26-a5c0-77783d8239b9-kube-api-access-g9tmn\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.899912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.899876 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.900125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.899932 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.900125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.900047 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.900296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.900274 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.900468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.900446 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.906524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.906221 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e4208b-1bbe-4a26-a5c0-77783d8239b9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.906524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.906402 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e4208b-1bbe-4a26-a5c0-77783d8239b9-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:50.909466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:50.909443 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9tmn\" (UniqueName: \"kubernetes.io/projected/96e4208b-1bbe-4a26-a5c0-77783d8239b9-kube-api-access-g9tmn\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl\" (UID: \"96e4208b-1bbe-4a26-a5c0-77783d8239b9\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:51.010003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:51.009918 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:51.297574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:51.297551 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl"] Nov 02 22:06:51.300080 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:06:51.300054 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e4208b_1bbe_4a26_a5c0_77783d8239b9.slice/crio-98b04d9e2ee22dba69bcf52d36f38f36f5b1d03c91029ede7948549182e277c6 WatchSource:0}: Error finding container 98b04d9e2ee22dba69bcf52d36f38f36f5b1d03c91029ede7948549182e277c6: Status 404 returned error can't find the container with id 98b04d9e2ee22dba69bcf52d36f38f36f5b1d03c91029ede7948549182e277c6 Nov 02 22:06:51.545341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:51.545306 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" event={"ID":"96e4208b-1bbe-4a26-a5c0-77783d8239b9","Type":"ContainerStarted","Data":"98b04d9e2ee22dba69bcf52d36f38f36f5b1d03c91029ede7948549182e277c6"} Nov 02 22:06:51.546859 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:51.546832 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" event={"ID":"774397e4-c8ee-4734-8f5a-7befd9b4d9dc","Type":"ContainerStarted","Data":"29a060efc2f892930c91748d40c148af723df0cd7ebe4c5638292cfef697399c"} Nov 02 22:06:51.564185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:51.564140 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" podStartSLOduration=1.814738124 podStartE2EDuration="3.564127783s" totalImagesPullingTime="1.749389659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:51.562745406 +0000 UTC m=+539.500858624" watchObservedRunningTime="2026-11-02 22:06:51.564127783 +0000 UTC m=+539.502241000" Nov 02 22:06:52.154405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:52.154376 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:52.553427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:52.553345 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" event={"ID":"96e4208b-1bbe-4a26-a5c0-77783d8239b9","Type":"ContainerStarted","Data":"512e125f3a9a3775c2491b8cba12e3bcfd771a121b9648a36ad649ffb778bab4"} Nov 02 22:06:52.571770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:52.571718 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" podStartSLOduration=2.299656944 podStartE2EDuration="2.571704449s" totalImagesPullingTime="272.047505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:06:52.568886053 +0000 UTC m=+540.506999273" watchObservedRunningTime="2026-11-02 22:06:52.571704449 +0000 UTC m=+540.509817667" Nov 02 22:06:53.010267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:53.010225 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:06:54.621204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:54.621174 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:54.634173 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:54.634147 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:55.148656 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:55.148620 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:55.161356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:55.161329 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:55.579305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:55.579274 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" Nov 02 22:06:55.584702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:55.584648 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" Nov 02 22:06:59.155139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:59.155107 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:59.167436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:59.167411 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:06:59.599744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:06:59.599718 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" Nov 02 22:07:01.010799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:01.010761 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:07:01.023248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:01.023219 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:07:01.609133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:01.609106 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" Nov 02 22:07:11.594472 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:07:11.594432 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:07:19.136174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.136140 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6"] Nov 02 22:07:19.146669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.146645 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.150206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.150179 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6"] Nov 02 22:07:19.153571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.153546 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 22:07:19.173877 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.173855 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9nj4\" (UniqueName: \"kubernetes.io/projected/f53ed47c-5e7c-4080-9452-63f49808674e-kube-api-access-b9nj4\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.174024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.173896 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.174024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.174007 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53ed47c-5e7c-4080-9452-63f49808674e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.174132 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.174065 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.174167 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.174142 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.174201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.174170 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.174201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.174190 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53ed47c-5e7c-4080-9452-63f49808674e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.174293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.174276 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53ed47c-5e7c-4080-9452-63f49808674e-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.174329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.174300 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.233302 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.233276 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mctrp\"" Nov 02 22:07:19.275159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275133 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53ed47c-5e7c-4080-9452-63f49808674e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275180 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275209 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275246 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275280 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53ed47c-5e7c-4080-9452-63f49808674e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275315 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53ed47c-5e7c-4080-9452-63f49808674e-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275354 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275402 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9nj4\" (UniqueName: \"kubernetes.io/projected/f53ed47c-5e7c-4080-9452-63f49808674e-kube-api-access-b9nj4\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275454 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275643 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275661 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275865 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.275961 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.275941 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.276091 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.276066 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53ed47c-5e7c-4080-9452-63f49808674e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.277646 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.277626 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53ed47c-5e7c-4080-9452-63f49808674e-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.277777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.277759 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53ed47c-5e7c-4080-9452-63f49808674e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.282319 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.282293 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53ed47c-5e7c-4080-9452-63f49808674e-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.282546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.282526 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9nj4\" (UniqueName: \"kubernetes.io/projected/f53ed47c-5e7c-4080-9452-63f49808674e-kube-api-access-b9nj4\") pod \"maas-default-gateway-openshift-default-696f47c6c-skqp6\" (UID: \"f53ed47c-5e7c-4080-9452-63f49808674e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.462568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.462489 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:19.745952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:19.740001 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6"] Nov 02 22:07:19.747377 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:07:19.747344 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53ed47c_5e7c_4080_9452_63f49808674e.slice/crio-60ddca15bd5cb657b49fac7e41106bae389b3fcb039bfb2fab022623322f051c WatchSource:0}: Error finding container 60ddca15bd5cb657b49fac7e41106bae389b3fcb039bfb2fab022623322f051c: Status 404 returned error can't find the container with id 60ddca15bd5cb657b49fac7e41106bae389b3fcb039bfb2fab022623322f051c Nov 02 22:07:20.688041 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:20.688005 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" event={"ID":"f53ed47c-5e7c-4080-9452-63f49808674e","Type":"ContainerStarted","Data":"60ddca15bd5cb657b49fac7e41106bae389b3fcb039bfb2fab022623322f051c"} Nov 02 22:07:23.039307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:23.039275 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:07:23.039612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:23.039340 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:07:23.039612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:23.039371 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:07:23.712875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:23.712839 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" event={"ID":"f53ed47c-5e7c-4080-9452-63f49808674e","Type":"ContainerStarted","Data":"91714ba9b34840de2e8cfeea2e61f0377ebb9542b9426ba6f846e148f7db6172"} Nov 02 22:07:23.733929 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:23.733880 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podStartSLOduration=1.4449144729999999 podStartE2EDuration="4.733864216s" totalImagesPullingTime="3.288949743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:23.732004234 +0000 UTC m=+571.670117452" watchObservedRunningTime="2026-11-02 22:07:23.733864216 +0000 UTC m=+571.671977433" Nov 02 22:07:24.462840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:24.462809 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:24.463922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:24.463900 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 22:07:24.463991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:24.463945 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 22:07:25.463173 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:25.463139 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 22:07:25.463547 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:25.463190 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 22:07:26.463384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:26.463346 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 22:07:26.463853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:26.463404 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 22:07:27.463819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:27.463783 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 22:07:27.464196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:27.463834 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 22:07:28.463437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:28.463290 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 22:07:28.463437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:28.463350 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 22:07:29.463656 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:29.463625 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 22:07:29.464065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:29.463701 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 22:07:30.463031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:30.462996 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Nov 02 22:07:30.463213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:30.463045 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Nov 02 22:07:31.615384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.615352 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-gl482"] Nov 02 22:07:31.621926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.621897 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:31.628279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.628245 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-gl482"] Nov 02 22:07:31.632373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.632346 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 22:07:31.632505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.632470 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 22:07:31.632590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.632524 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xq9vk\"" Nov 02 22:07:31.632684 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.632656 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 22:07:31.633142 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.633125 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 22:07:31.690563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.690538 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f6rg\" (UniqueName: \"kubernetes.io/projected/94d248dc-8d82-4f2c-add3-aa80d9685c10-kube-api-access-5f6rg\") pod \"ai-gateway-controller-55cd769764-gl482\" (UID: \"94d248dc-8d82-4f2c-add3-aa80d9685c10\") " pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:31.690711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.690592 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d248dc-8d82-4f2c-add3-aa80d9685c10-metrics-certs\") pod \"ai-gateway-controller-55cd769764-gl482\" (UID: \"94d248dc-8d82-4f2c-add3-aa80d9685c10\") " pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:31.792132 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.792099 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f6rg\" (UniqueName: \"kubernetes.io/projected/94d248dc-8d82-4f2c-add3-aa80d9685c10-kube-api-access-5f6rg\") pod \"ai-gateway-controller-55cd769764-gl482\" (UID: \"94d248dc-8d82-4f2c-add3-aa80d9685c10\") " pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:31.792322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.792170 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d248dc-8d82-4f2c-add3-aa80d9685c10-metrics-certs\") pod \"ai-gateway-controller-55cd769764-gl482\" (UID: \"94d248dc-8d82-4f2c-add3-aa80d9685c10\") " pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:31.794666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.794640 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d248dc-8d82-4f2c-add3-aa80d9685c10-metrics-certs\") pod \"ai-gateway-controller-55cd769764-gl482\" (UID: \"94d248dc-8d82-4f2c-add3-aa80d9685c10\") " pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:31.799728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.799703 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f6rg\" (UniqueName: \"kubernetes.io/projected/94d248dc-8d82-4f2c-add3-aa80d9685c10-kube-api-access-5f6rg\") pod \"ai-gateway-controller-55cd769764-gl482\" (UID: \"94d248dc-8d82-4f2c-add3-aa80d9685c10\") " pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:31.941634 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:31.941555 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:32.217747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:32.217717 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-gl482"] Nov 02 22:07:32.219822 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:07:32.219787 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d248dc_8d82_4f2c_add3_aa80d9685c10.slice/crio-4019500919b2de0eda47e65f37749803928edd35d4d42ad8678f7b151f04205f WatchSource:0}: Error finding container 4019500919b2de0eda47e65f37749803928edd35d4d42ad8678f7b151f04205f: Status 404 returned error can't find the container with id 4019500919b2de0eda47e65f37749803928edd35d4d42ad8678f7b151f04205f Nov 02 22:07:32.505929 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:32.505830 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:07:32.505929 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:32.505910 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:07:32.759638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:32.759554 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" event={"ID":"94d248dc-8d82-4f2c-add3-aa80d9685c10","Type":"ContainerStarted","Data":"4019500919b2de0eda47e65f37749803928edd35d4d42ad8678f7b151f04205f"} Nov 02 22:07:33.537648 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:33.537614 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:07:33.537839 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:33.537686 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:07:33.544035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:33.544008 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:33.544396 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:33.544377 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:33.545281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:33.545261 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" Nov 02 22:07:34.147692 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.147644 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6"] Nov 02 22:07:34.154316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.154291 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.174012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.173894 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6"] Nov 02 22:07:34.217548 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.217512 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.217758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.217572 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45cht\" (UniqueName: \"kubernetes.io/projected/e9c4918d-e8b3-4e8a-a8ee-651b35456661-kube-api-access-45cht\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.217758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.217670 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.217758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.217730 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.217758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.217756 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.217983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.217906 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.217983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.217954 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.218081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.218020 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.218124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.218101 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.319724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.319669 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.319900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.319771 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.319900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.319831 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45cht\" (UniqueName: \"kubernetes.io/projected/e9c4918d-e8b3-4e8a-a8ee-651b35456661-kube-api-access-45cht\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.319900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.319884 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.319916 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.319939 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.319997 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.320029 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.320057 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.320067 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.320387 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.320455 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.320784 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.320873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.320815 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.322553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.322522 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.322914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.322896 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.328116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.328090 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45cht\" (UniqueName: \"kubernetes.io/projected/e9c4918d-e8b3-4e8a-a8ee-651b35456661-kube-api-access-45cht\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.328266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.328244 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9c4918d-e8b3-4e8a-a8ee-651b35456661-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-xlsg6\" (UID: \"e9c4918d-e8b3-4e8a-a8ee-651b35456661\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.493737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.493629 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:34.788256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.788223 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6"] Nov 02 22:07:34.792098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.792058 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:07:34.792225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.792144 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:07:34.792225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:34.792186 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:07:35.778902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:35.778805 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" event={"ID":"e9c4918d-e8b3-4e8a-a8ee-651b35456661","Type":"ContainerStarted","Data":"e443adb5a820bef955e912499d5e8e1f01057134ff90964a98e3d1f77cebc432"} Nov 02 22:07:35.778902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:35.778889 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" event={"ID":"e9c4918d-e8b3-4e8a-a8ee-651b35456661","Type":"ContainerStarted","Data":"650c5a92336f8054a54ffee23e3f3afbf695b57420608eefe7600ae174b0f7e6"} Nov 02 22:07:35.780300 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:35.780272 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" event={"ID":"94d248dc-8d82-4f2c-add3-aa80d9685c10","Type":"ContainerStarted","Data":"986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7"} Nov 02 22:07:35.780415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:35.780322 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:35.814699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:35.814458 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podStartSLOduration=1.814439068 podStartE2EDuration="1.814439068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:35.805576689 +0000 UTC m=+583.743689908" watchObservedRunningTime="2026-11-02 22:07:35.814439068 +0000 UTC m=+583.752552287" Nov 02 22:07:35.827711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:35.827646 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" podStartSLOduration=1.702857336 podStartE2EDuration="4.827630952s" totalImagesPullingTime="3.124773616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:35.826071003 +0000 UTC m=+583.764184215" watchObservedRunningTime="2026-11-02 22:07:35.827630952 +0000 UTC m=+583.765744169" Nov 02 22:07:36.261349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.261314 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6"] Nov 02 22:07:36.265404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.265382 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:36.271253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.271228 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pgv9t\"" Nov 02 22:07:36.271377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.271271 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 22:07:36.274332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.273254 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6"] Nov 02 22:07:36.341667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.341627 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ac817a-ffba-42a8-b379-1fa39f0f81da-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qhdt6\" (UID: \"20ac817a-ffba-42a8-b379-1fa39f0f81da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:36.341845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.341797 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j9df\" (UniqueName: \"kubernetes.io/projected/20ac817a-ffba-42a8-b379-1fa39f0f81da-kube-api-access-9j9df\") pod \"payload-pre-processing-7b98bb678c-qhdt6\" (UID: \"20ac817a-ffba-42a8-b379-1fa39f0f81da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:36.442762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.442728 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j9df\" (UniqueName: \"kubernetes.io/projected/20ac817a-ffba-42a8-b379-1fa39f0f81da-kube-api-access-9j9df\") pod \"payload-pre-processing-7b98bb678c-qhdt6\" (UID: \"20ac817a-ffba-42a8-b379-1fa39f0f81da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:36.442983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.442876 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ac817a-ffba-42a8-b379-1fa39f0f81da-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qhdt6\" (UID: \"20ac817a-ffba-42a8-b379-1fa39f0f81da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:36.443789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.443765 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20ac817a-ffba-42a8-b379-1fa39f0f81da-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qhdt6\" (UID: \"20ac817a-ffba-42a8-b379-1fa39f0f81da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:36.450172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.450132 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j9df\" (UniqueName: \"kubernetes.io/projected/20ac817a-ffba-42a8-b379-1fa39f0f81da-kube-api-access-9j9df\") pod \"payload-pre-processing-7b98bb678c-qhdt6\" (UID: \"20ac817a-ffba-42a8-b379-1fa39f0f81da\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:36.499699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.499020 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:36.504699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.502182 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xlsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:07:36.504699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.502235 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podUID="e9c4918d-e8b3-4e8a-a8ee-651b35456661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 22:07:36.580708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.580643 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:36.901751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:36.900802 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6"] Nov 02 22:07:36.915152 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:07:36.915114 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ac817a_ffba_42a8_b379_1fa39f0f81da.slice/crio-950b973035dd4787f42e39bd1cfc5253a10bbd7993eabb9ce270d6756c6db3f3 WatchSource:0}: Error finding container 950b973035dd4787f42e39bd1cfc5253a10bbd7993eabb9ce270d6756c6db3f3: Status 404 returned error can't find the container with id 950b973035dd4787f42e39bd1cfc5253a10bbd7993eabb9ce270d6756c6db3f3 Nov 02 22:07:37.497473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:37.497017 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xlsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:07:37.497473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:37.497091 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podUID="e9c4918d-e8b3-4e8a-a8ee-651b35456661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 22:07:37.826262 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:37.826190 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" event={"ID":"20ac817a-ffba-42a8-b379-1fa39f0f81da","Type":"ContainerStarted","Data":"950b973035dd4787f42e39bd1cfc5253a10bbd7993eabb9ce270d6756c6db3f3"} Nov 02 22:07:38.500546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:38.500106 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xlsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:07:38.500546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:38.500166 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podUID="e9c4918d-e8b3-4e8a-a8ee-651b35456661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 22:07:39.507715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:39.505082 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xlsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:07:39.507715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:39.505143 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podUID="e9c4918d-e8b3-4e8a-a8ee-651b35456661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 22:07:40.500701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:40.497126 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xlsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:07:40.500701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:40.497191 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podUID="e9c4918d-e8b3-4e8a-a8ee-651b35456661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 22:07:40.842612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:40.842574 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" event={"ID":"20ac817a-ffba-42a8-b379-1fa39f0f81da","Type":"ContainerStarted","Data":"4f62ec5bfc544337a7eb685f2a311e20d5f9f8766d3606255cc8da03e52189b9"} Nov 02 22:07:40.843018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:40.842734 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:40.859519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:40.859468 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" podStartSLOduration=1.918183391 podStartE2EDuration="4.859443997s" totalImagesPullingTime="2.941260606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:40.857730046 +0000 UTC m=+588.795843264" watchObservedRunningTime="2026-11-02 22:07:40.859443997 +0000 UTC m=+588.797557215" Nov 02 22:07:41.494171 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:41.494140 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xlsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:07:41.494341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:41.494194 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podUID="e9c4918d-e8b3-4e8a-a8ee-651b35456661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 22:07:41.595217 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:07:41.595182 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:07:42.494947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:42.494911 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xlsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 22:07:42.495327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:42.494962 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podUID="e9c4918d-e8b3-4e8a-a8ee-651b35456661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 22:07:44.535847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:44.535815 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xlsg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:07:44.536369 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:44.535879 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podUID="e9c4918d-e8b3-4e8a-a8ee-651b35456661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:07:45.314832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:45.314803 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:45.315570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:45.315545 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:45.316162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:45.316147 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" Nov 02 22:07:46.816667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:46.816639 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:07:47.071994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.071911 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr"] Nov 02 22:07:47.075815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.075791 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:47.083064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.083039 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr"] Nov 02 22:07:47.183093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.183055 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfhj\" (UniqueName: \"kubernetes.io/projected/b7f87410-f023-4314-b687-5fac8d1922f4-kube-api-access-qlfhj\") pod \"ai-gateway-controller-7d64dfbf7-vm8gr\" (UID: \"b7f87410-f023-4314-b687-5fac8d1922f4\") " pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:47.183273 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.183104 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f87410-f023-4314-b687-5fac8d1922f4-metrics-certs\") pod \"ai-gateway-controller-7d64dfbf7-vm8gr\" (UID: \"b7f87410-f023-4314-b687-5fac8d1922f4\") " pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:47.283989 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.283953 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfhj\" (UniqueName: \"kubernetes.io/projected/b7f87410-f023-4314-b687-5fac8d1922f4-kube-api-access-qlfhj\") pod \"ai-gateway-controller-7d64dfbf7-vm8gr\" (UID: \"b7f87410-f023-4314-b687-5fac8d1922f4\") " pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:47.284328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.284298 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f87410-f023-4314-b687-5fac8d1922f4-metrics-certs\") pod \"ai-gateway-controller-7d64dfbf7-vm8gr\" (UID: \"b7f87410-f023-4314-b687-5fac8d1922f4\") " pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:47.286578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.286555 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7f87410-f023-4314-b687-5fac8d1922f4-metrics-certs\") pod \"ai-gateway-controller-7d64dfbf7-vm8gr\" (UID: \"b7f87410-f023-4314-b687-5fac8d1922f4\") " pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:47.291768 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.291745 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfhj\" (UniqueName: \"kubernetes.io/projected/b7f87410-f023-4314-b687-5fac8d1922f4-kube-api-access-qlfhj\") pod \"ai-gateway-controller-7d64dfbf7-vm8gr\" (UID: \"b7f87410-f023-4314-b687-5fac8d1922f4\") " pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:47.392081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.392004 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:47.663584 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.663559 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr"] Nov 02 22:07:47.666527 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:07:47.666498 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f87410_f023_4314_b687_5fac8d1922f4.slice/crio-c17e4933c2b5202ad17382dfa2d715bb21526dba34fbf6f35247e491ef0f0630 WatchSource:0}: Error finding container c17e4933c2b5202ad17382dfa2d715bb21526dba34fbf6f35247e491ef0f0630: Status 404 returned error can't find the container with id c17e4933c2b5202ad17382dfa2d715bb21526dba34fbf6f35247e491ef0f0630 Nov 02 22:07:47.878892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:47.878860 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" event={"ID":"b7f87410-f023-4314-b687-5fac8d1922f4","Type":"ContainerStarted","Data":"c17e4933c2b5202ad17382dfa2d715bb21526dba34fbf6f35247e491ef0f0630"} Nov 02 22:07:48.885246 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:48.885204 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" event={"ID":"b7f87410-f023-4314-b687-5fac8d1922f4","Type":"ContainerStarted","Data":"ac423904ba80ffd9235cec984117ba35d4aeab6c18af9b4bce86a5959ec5e3e2"} Nov 02 22:07:48.885621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:48.885337 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:48.901975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:48.901933 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" podStartSLOduration=1.351880914 podStartE2EDuration="1.901918331s" totalImagesPullingTime="550.037417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:07:48.900071959 +0000 UTC m=+596.838185178" watchObservedRunningTime="2026-11-02 22:07:48.901918331 +0000 UTC m=+596.840031591" Nov 02 22:07:51.853810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:51.853781 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qhdt6" Nov 02 22:07:52.227588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:52.227515 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:07:52.228730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:52.228706 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:07:59.892773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:59.892737 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d64dfbf7-vm8gr" Nov 02 22:07:59.938618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:59.938585 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-gl482"] Nov 02 22:07:59.938932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:07:59.938899 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" podUID="94d248dc-8d82-4f2c-add3-aa80d9685c10" containerName="manager" containerID="cri-o://986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7" gracePeriod=30 Nov 02 22:08:00.183709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.183683 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:08:00.204373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.204347 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d248dc-8d82-4f2c-add3-aa80d9685c10-kube-api-access-5f6rg\" (UniqueName: \"kubernetes.io/projected/94d248dc-8d82-4f2c-add3-aa80d9685c10-kube-api-access-5f6rg\") pod \"94d248dc-8d82-4f2c-add3-aa80d9685c10\" (UID: \"94d248dc-8d82-4f2c-add3-aa80d9685c10\") " Nov 02 22:08:00.204523 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.204417 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94d248dc-8d82-4f2c-add3-aa80d9685c10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d248dc-8d82-4f2c-add3-aa80d9685c10-metrics-certs\") pod \"94d248dc-8d82-4f2c-add3-aa80d9685c10\" (UID: \"94d248dc-8d82-4f2c-add3-aa80d9685c10\") " Nov 02 22:08:00.207256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.207226 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d248dc-8d82-4f2c-add3-aa80d9685c10-kube-api-access-5f6rg" pod "94d248dc-8d82-4f2c-add3-aa80d9685c10" (UID: "94d248dc-8d82-4f2c-add3-aa80d9685c10"). InnerVolumeSpecName "kube-api-access-5f6rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:08:00.207256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.207241 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d248dc-8d82-4f2c-add3-aa80d9685c10-metrics-certs" pod "94d248dc-8d82-4f2c-add3-aa80d9685c10" (UID: "94d248dc-8d82-4f2c-add3-aa80d9685c10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:08:00.305828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.305793 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f6rg\" (UniqueName: \"kubernetes.io/projected/94d248dc-8d82-4f2c-add3-aa80d9685c10-kube-api-access-5f6rg\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:00.305828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.305827 4309 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94d248dc-8d82-4f2c-add3-aa80d9685c10-metrics-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:08:00.941405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.941361 4309 generic.go:356] "Generic (PLEG): container finished" podID="94d248dc-8d82-4f2c-add3-aa80d9685c10" containerID="986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7" exitCode=0 Nov 02 22:08:00.941841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.941411 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" Nov 02 22:08:00.941841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.941435 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" event={"ID":"94d248dc-8d82-4f2c-add3-aa80d9685c10","Type":"ContainerDied","Data":"986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7"} Nov 02 22:08:00.941841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.941518 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-55cd769764-gl482" event={"ID":"94d248dc-8d82-4f2c-add3-aa80d9685c10","Type":"ContainerDied","Data":"4019500919b2de0eda47e65f37749803928edd35d4d42ad8678f7b151f04205f"} Nov 02 22:08:00.941841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.941552 4309 scope.go:122] "RemoveContainer" containerID="986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7" Nov 02 22:08:00.950760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.950742 4309 scope.go:122] "RemoveContainer" containerID="986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7" Nov 02 22:08:00.951035 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:08:00.951017 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7\": container with ID starting with 986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7 not found: ID does not exist" containerID="986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7" Nov 02 22:08:00.951083 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.951043 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7"} err="failed to get container status \"986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7\": rpc error: code = NotFound desc = could not find container \"986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7\": container with ID starting with 986d6d4813b36f83d9e9d35ec9d28bd86e5e751e8b1751dd97e770eed0d2b2a7 not found: ID does not exist" Nov 02 22:08:00.962937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.962915 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-gl482"] Nov 02 22:08:00.964820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:00.964801 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-55cd769764-gl482"] Nov 02 22:08:02.187267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:02.187237 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d248dc-8d82-4f2c-add3-aa80d9685c10" path="/var/lib/kubelet/pods/94d248dc-8d82-4f2c-add3-aa80d9685c10/volumes" Nov 02 22:08:11.595070 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:08:11.595033 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:08:29.033586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.033548 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd"] Nov 02 22:08:29.034195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.034149 4309 cpu_manager.go:414] "containerMap: removing container" podUID="94d248dc-8d82-4f2c-add3-aa80d9685c10" containerName="manager" Nov 02 22:08:29.034195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.034166 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d248dc-8d82-4f2c-add3-aa80d9685c10" containerName="manager" Nov 02 22:08:29.034418 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.034350 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d248dc-8d82-4f2c-add3-aa80d9685c10" containerName="manager" Nov 02 22:08:29.041045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.041023 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.052464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.052439 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd"] Nov 02 22:08:29.068785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.068764 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5f4a2f-openshift-default-dockercfg-d97zb\"" Nov 02 22:08:29.121759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.121728 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p"] Nov 02 22:08:29.125368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.125343 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.136870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.136846 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p"] Nov 02 22:08:29.153882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.153859 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-719012-openshift-default-dockercfg-glgls\"" Nov 02 22:08:29.191975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.191932 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.192107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.192075 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.192178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.192157 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx4l6\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-kube-api-access-wx4l6\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.192217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.192203 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.192274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.192246 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.192315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.192287 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.192358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.192315 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.192412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.192396 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.192469 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.192452 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.293836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.293738 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.294100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294081 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.294205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294189 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.294277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294223 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.294277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294146 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.294277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294252 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.294440 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294345 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.294495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294450 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.294556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294489 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.294611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294584 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.294646 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294614 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx4l6\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-kube-api-access-wx4l6\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294650 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294717 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n8gb\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-kube-api-access-8n8gb\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.294759 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.295126 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.295169 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw"] Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.295164 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.295217 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.295272 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.295306 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.295809 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.296042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.295929 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.296640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.296057 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.296640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.296082 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.297552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.297506 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.298593 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.298564 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.300200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.300181 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.302945 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.302921 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.306539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.306520 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx4l6\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-kube-api-access-wx4l6\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.314526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.314500 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw"] Nov 02 22:08:29.314812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.314782 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a1d78c-openshift-default-dockercfg-dr8jk\"" Nov 02 22:08:29.357194 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.357163 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:29.396804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.396766 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.396965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.396818 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.396982 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.397235 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.397265 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.397265 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.397304 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.397387 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397566 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.397449 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n8gb\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-kube-api-access-8n8gb\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397566 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.397493 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397566 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.397542 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.397993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.397947 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.398198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.398160 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.398504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.398477 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.400307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.400284 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.400515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.400495 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.408293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.408265 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.409120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.409096 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n8gb\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-kube-api-access-8n8gb\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.441237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.441202 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:29.473693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.471950 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f"] Nov 02 22:08:29.477370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.477148 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.487281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.486815 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c302b6-openshift-default-dockercfg-wc6gv\"" Nov 02 22:08:29.493065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.492842 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f"] Nov 02 22:08:29.499474 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.499440 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.499735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.499505 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.499735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.499550 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.499735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.499620 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f302bd-e66a-43b9-b818-8748b284d98f-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.499735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.499663 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.499735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.499711 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.500146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.499744 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.500552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500523 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.500649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500575 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-kube-api-access-swvvb\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.500649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500619 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.500799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500647 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639a7482-b746-42ea-a91e-95b7a39cd269-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.500799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500689 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.500799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500740 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f302bd-e66a-43b9-b818-8748b284d98f-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.500799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500781 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.500973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500815 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639a7482-b746-42ea-a91e-95b7a39cd269-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.500973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500881 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.500973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500907 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hhh2\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-kube-api-access-4hhh2\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.500973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.500937 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.602187 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602157 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.602375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602203 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639a7482-b746-42ea-a91e-95b7a39cd269-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.602375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602259 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.602375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602284 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhh2\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-kube-api-access-4hhh2\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.602375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602310 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.602375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602351 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602401 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602430 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602459 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f302bd-e66a-43b9-b818-8748b284d98f-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602507 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602534 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602571 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602587 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602614 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602645 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-kube-api-access-swvvb\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.602711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602707 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.603239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602737 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639a7482-b746-42ea-a91e-95b7a39cd269-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.603239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602765 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.603239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602822 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f302bd-e66a-43b9-b818-8748b284d98f-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.603239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.602872 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.603239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.603113 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.603630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.603602 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f302bd-e66a-43b9-b818-8748b284d98f-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.603786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.603667 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639a7482-b746-42ea-a91e-95b7a39cd269-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.604003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.603953 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.604697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.604131 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.604697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.604413 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.604697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.604616 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.605059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.605040 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.605460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.605435 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.605861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.605845 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f302bd-e66a-43b9-b818-8748b284d98f-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.606534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.606511 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639a7482-b746-42ea-a91e-95b7a39cd269-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.607653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.607632 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.613194 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.613169 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-kube-api-access-swvvb\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.613349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.613324 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.613473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.613453 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.613786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.613769 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhh2\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-kube-api-access-4hhh2\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.634779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.634749 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:29.643521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.643501 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd"] Nov 02 22:08:29.647703 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:29.647631 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a6102b_8f0b_4ecc_8e29_836c57f0420b.slice/crio-30e6177e4696ceeed8ef6c195334bd187d89fd8766bf98644b907ad7ed5da3e0 WatchSource:0}: Error finding container 30e6177e4696ceeed8ef6c195334bd187d89fd8766bf98644b907ad7ed5da3e0: Status 404 returned error can't find the container with id 30e6177e4696ceeed8ef6c195334bd187d89fd8766bf98644b907ad7ed5da3e0 Nov 02 22:08:29.650300 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.650268 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:29.650374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.650346 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:29.650417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.650385 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:29.744846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.744331 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p"] Nov 02 22:08:29.749423 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:29.749337 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ff0346_d7be_4890_bdfe_7c7b9d398fd5.slice/crio-04badde7c7f1c08c5091fca2914819489375c0211e5968d791a750bea8edacc4 WatchSource:0}: Error finding container 04badde7c7f1c08c5091fca2914819489375c0211e5968d791a750bea8edacc4: Status 404 returned error can't find the container with id 04badde7c7f1c08c5091fca2914819489375c0211e5968d791a750bea8edacc4 Nov 02 22:08:29.755802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.755761 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:29.755916 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.755853 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:29.755916 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.755901 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:29.800094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.800062 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:29.926946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.926900 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw"] Nov 02 22:08:29.942127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.942093 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:29.942232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.942178 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:29.942232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:29.942220 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:30.087080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.085537 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" event={"ID":"639a7482-b746-42ea-a91e-95b7a39cd269","Type":"ContainerStarted","Data":"9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d"} Nov 02 22:08:30.087080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.085601 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" event={"ID":"639a7482-b746-42ea-a91e-95b7a39cd269","Type":"ContainerStarted","Data":"8bd2a61e2724e9bf07c9c6022b17301eb80fa5853d266845e3afdffbad87db4f"} Nov 02 22:08:30.090285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.090257 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" event={"ID":"95a6102b-8f0b-4ecc-8e29-836c57f0420b","Type":"ContainerStarted","Data":"afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e"} Nov 02 22:08:30.092339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.092314 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" event={"ID":"95a6102b-8f0b-4ecc-8e29-836c57f0420b","Type":"ContainerStarted","Data":"30e6177e4696ceeed8ef6c195334bd187d89fd8766bf98644b907ad7ed5da3e0"} Nov 02 22:08:30.092794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.092766 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" event={"ID":"66ff0346-d7be-4890-bdfe-7c7b9d398fd5","Type":"ContainerStarted","Data":"ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035"} Nov 02 22:08:30.092888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.092806 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" event={"ID":"66ff0346-d7be-4890-bdfe-7c7b9d398fd5","Type":"ContainerStarted","Data":"04badde7c7f1c08c5091fca2914819489375c0211e5968d791a750bea8edacc4"} Nov 02 22:08:30.107043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.106987 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podStartSLOduration=1.10696859 podStartE2EDuration="1.10696859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:30.102732 +0000 UTC m=+638.040845219" watchObservedRunningTime="2026-11-02 22:08:30.10696859 +0000 UTC m=+638.045081812" Nov 02 22:08:30.110780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.110574 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f"] Nov 02 22:08:30.115651 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:30.115610 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f302bd_e66a_43b9_b818_8748b284d98f.slice/crio-39a4623af600b77711feda88daf2e10b3904a62fa777bddbc813bdb1cfa28f7a WatchSource:0}: Error finding container 39a4623af600b77711feda88daf2e10b3904a62fa777bddbc813bdb1cfa28f7a: Status 404 returned error can't find the container with id 39a4623af600b77711feda88daf2e10b3904a62fa777bddbc813bdb1cfa28f7a Nov 02 22:08:30.118043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.118015 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:30.118155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.118075 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:30.118155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.118106 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:30.149887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.149837 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" podStartSLOduration=1.14981833 podStartE2EDuration="1.14981833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:30.123767397 +0000 UTC m=+638.061880617" watchObservedRunningTime="2026-11-02 22:08:30.14981833 +0000 UTC m=+638.087931551" Nov 02 22:08:30.150016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.149991 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podStartSLOduration=1.149984645 podStartE2EDuration="1.149984645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:30.147017569 +0000 UTC m=+638.085130784" watchObservedRunningTime="2026-11-02 22:08:30.149984645 +0000 UTC m=+638.088097866" Nov 02 22:08:30.358403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.358376 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:30.363543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.363518 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:30.442290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.442207 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:30.447565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.447537 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:30.635010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:30.634973 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:31.099116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.099080 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" event={"ID":"83f302bd-e66a-43b9-b818-8748b284d98f","Type":"ContainerStarted","Data":"1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c"} Nov 02 22:08:31.099574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.099135 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" event={"ID":"83f302bd-e66a-43b9-b818-8748b284d98f","Type":"ContainerStarted","Data":"39a4623af600b77711feda88daf2e10b3904a62fa777bddbc813bdb1cfa28f7a"} Nov 02 22:08:31.100051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.099743 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:31.100051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.099781 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:31.100532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.100509 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:08:31.100658 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.100642 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:31.119905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.119853 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podStartSLOduration=2.119841242 podStartE2EDuration="2.119841242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:31.117661529 +0000 UTC m=+639.055774748" watchObservedRunningTime="2026-11-02 22:08:31.119841242 +0000 UTC m=+639.057954459" Nov 02 22:08:31.640983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.640954 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:31.800835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.800800 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:31.806213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:31.806190 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:32.104527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:32.104499 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:32.105051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:32.104736 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:32.105564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:32.105543 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:08:32.105732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:32.105715 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:34.444386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.444352 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph"] Nov 02 22:08:34.448130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.448114 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:34.455143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.455102 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-719012\"" Nov 02 22:08:34.460350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.460324 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph"] Nov 02 22:08:34.561832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.561790 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7315bae5-39b1-4241-bba0-303c9a5ad5e7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph\" (UID: \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:34.562022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.561905 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfsrb\" (UniqueName: \"kubernetes.io/projected/7315bae5-39b1-4241-bba0-303c9a5ad5e7-kube-api-access-pfsrb\") pod \"payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph\" (UID: \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:34.663369 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.663334 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7315bae5-39b1-4241-bba0-303c9a5ad5e7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph\" (UID: \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:34.663553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.663478 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfsrb\" (UniqueName: \"kubernetes.io/projected/7315bae5-39b1-4241-bba0-303c9a5ad5e7-kube-api-access-pfsrb\") pod \"payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph\" (UID: \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:34.664185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.664155 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7315bae5-39b1-4241-bba0-303c9a5ad5e7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph\" (UID: \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:34.673649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.673621 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfsrb\" (UniqueName: \"kubernetes.io/projected/7315bae5-39b1-4241-bba0-303c9a5ad5e7-kube-api-access-pfsrb\") pod \"payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph\" (UID: \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:34.764305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:34.764221 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:35.028731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.028632 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6"] Nov 02 22:08:35.033549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.033526 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:35.042047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.042003 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5f4a2f-dockercfg-62lkr\"" Nov 02 22:08:35.047218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.047188 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6"] Nov 02 22:08:35.048361 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:35.048327 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7315bae5_39b1_4241_bba0_303c9a5ad5e7.slice/crio-d2364e719d8b292f75f0a8e9852cacd9b7561c8d937042102f863611bef32763 WatchSource:0}: Error finding container d2364e719d8b292f75f0a8e9852cacd9b7561c8d937042102f863611bef32763: Status 404 returned error can't find the container with id d2364e719d8b292f75f0a8e9852cacd9b7561c8d937042102f863611bef32763 Nov 02 22:08:35.050260 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.050227 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph"] Nov 02 22:08:35.068931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.068865 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f6d309-4012-493e-a01f-c2e3acfbf6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6\" (UID: \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:35.069102 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.068991 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/13f6d309-4012-493e-a01f-c2e3acfbf6a5-kube-api-access-mxcm2\") pod \"payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6\" (UID: \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:35.100309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.100285 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5f4a2f\"" Nov 02 22:08:35.121851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.121825 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" event={"ID":"7315bae5-39b1-4241-bba0-303c9a5ad5e7","Type":"ContainerStarted","Data":"e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b"} Nov 02 22:08:35.121950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.121877 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" event={"ID":"7315bae5-39b1-4241-bba0-303c9a5ad5e7","Type":"ContainerStarted","Data":"d2364e719d8b292f75f0a8e9852cacd9b7561c8d937042102f863611bef32763"} Nov 02 22:08:35.121950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.121910 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:35.140227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.140183 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" podStartSLOduration=1.140168643 podStartE2EDuration="1.140168643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:35.138052092 +0000 UTC m=+643.076165301" watchObservedRunningTime="2026-11-02 22:08:35.140168643 +0000 UTC m=+643.078281877" Nov 02 22:08:35.169653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.169623 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/13f6d309-4012-493e-a01f-c2e3acfbf6a5-kube-api-access-mxcm2\") pod \"payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6\" (UID: \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:35.169814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.169730 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f6d309-4012-493e-a01f-c2e3acfbf6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6\" (UID: \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:35.170606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.170582 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f6d309-4012-493e-a01f-c2e3acfbf6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6\" (UID: \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:35.178362 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.178344 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/13f6d309-4012-493e-a01f-c2e3acfbf6a5-kube-api-access-mxcm2\") pod \"payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6\" (UID: \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:35.347868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.347833 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:35.592534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.592498 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5"] Nov 02 22:08:35.597253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.597231 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:35.605021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.604212 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2532af-dockercfg-m6sd4\"" Nov 02 22:08:35.605021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.604233 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2532af\"" Nov 02 22:08:35.606807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.606782 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5"] Nov 02 22:08:35.630325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.630296 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6"] Nov 02 22:08:35.635074 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:35.635049 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f6d309_4012_493e_a01f_c2e3acfbf6a5.slice/crio-545982258ff2898a3f12062e9d00e6b91f4b7893c5da8a2d300fb4d192b9c7a1 WatchSource:0}: Error finding container 545982258ff2898a3f12062e9d00e6b91f4b7893c5da8a2d300fb4d192b9c7a1: Status 404 returned error can't find the container with id 545982258ff2898a3f12062e9d00e6b91f4b7893c5da8a2d300fb4d192b9c7a1 Nov 02 22:08:35.655060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.654931 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2"] Nov 02 22:08:35.660313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.660292 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:35.667597 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.667573 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2"] Nov 02 22:08:35.676298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.675970 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b0dad1-c312-47fd-ac10-8b4d26a32735-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5\" (UID: \"08b0dad1-c312-47fd-ac10-8b4d26a32735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:35.676298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.676170 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24mv8\" (UniqueName: \"kubernetes.io/projected/08b0dad1-c312-47fd-ac10-8b4d26a32735-kube-api-access-24mv8\") pod \"payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5\" (UID: \"08b0dad1-c312-47fd-ac10-8b4d26a32735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:35.777476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.777445 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90333f30-7fb2-446f-b2b4-a9c2792c5794-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2\" (UID: \"90333f30-7fb2-446f-b2b4-a9c2792c5794\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:35.777613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.777504 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24mv8\" (UniqueName: \"kubernetes.io/projected/08b0dad1-c312-47fd-ac10-8b4d26a32735-kube-api-access-24mv8\") pod \"payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5\" (UID: \"08b0dad1-c312-47fd-ac10-8b4d26a32735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:35.777662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.777613 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s92r\" (UniqueName: \"kubernetes.io/projected/90333f30-7fb2-446f-b2b4-a9c2792c5794-kube-api-access-5s92r\") pod \"payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2\" (UID: \"90333f30-7fb2-446f-b2b4-a9c2792c5794\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:35.777777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.777760 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b0dad1-c312-47fd-ac10-8b4d26a32735-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5\" (UID: \"08b0dad1-c312-47fd-ac10-8b4d26a32735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:35.778446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.778429 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b0dad1-c312-47fd-ac10-8b4d26a32735-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5\" (UID: \"08b0dad1-c312-47fd-ac10-8b4d26a32735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:35.785153 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.785122 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24mv8\" (UniqueName: \"kubernetes.io/projected/08b0dad1-c312-47fd-ac10-8b4d26a32735-kube-api-access-24mv8\") pod \"payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5\" (UID: \"08b0dad1-c312-47fd-ac10-8b4d26a32735\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:35.879865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.879722 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90333f30-7fb2-446f-b2b4-a9c2792c5794-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2\" (UID: \"90333f30-7fb2-446f-b2b4-a9c2792c5794\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:35.881366 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.881244 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90333f30-7fb2-446f-b2b4-a9c2792c5794-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2\" (UID: \"90333f30-7fb2-446f-b2b4-a9c2792c5794\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:35.881505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.881395 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s92r\" (UniqueName: \"kubernetes.io/projected/90333f30-7fb2-446f-b2b4-a9c2792c5794-kube-api-access-5s92r\") pod \"payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2\" (UID: \"90333f30-7fb2-446f-b2b4-a9c2792c5794\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:35.890510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.890482 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s92r\" (UniqueName: \"kubernetes.io/projected/90333f30-7fb2-446f-b2b4-a9c2792c5794-kube-api-access-5s92r\") pod \"payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2\" (UID: \"90333f30-7fb2-446f-b2b4-a9c2792c5794\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:35.916017 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:35.915987 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:36.026267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.026227 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:36.130858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.130778 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" event={"ID":"13f6d309-4012-493e-a01f-c2e3acfbf6a5","Type":"ContainerStarted","Data":"e6079a91d58ed7d37a2eec9e79403ad53e16c26acbbf7c3752dc4d65aa66f6ba"} Nov 02 22:08:36.131000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.130916 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:36.131000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.130940 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" event={"ID":"13f6d309-4012-493e-a01f-c2e3acfbf6a5","Type":"ContainerStarted","Data":"545982258ff2898a3f12062e9d00e6b91f4b7893c5da8a2d300fb4d192b9c7a1"} Nov 02 22:08:36.149718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.149648 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" podStartSLOduration=1.149634953 podStartE2EDuration="1.149634953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:36.14811815 +0000 UTC m=+644.086231369" watchObservedRunningTime="2026-11-02 22:08:36.149634953 +0000 UTC m=+644.087748170" Nov 02 22:08:36.204744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.204721 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5"] Nov 02 22:08:36.206149 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:36.206121 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b0dad1_c312_47fd_ac10_8b4d26a32735.slice/crio-a817287dad1cf7fa2edceb42143dfb8692647ee2d1fc151406b27e4a7f72bf88 WatchSource:0}: Error finding container a817287dad1cf7fa2edceb42143dfb8692647ee2d1fc151406b27e4a7f72bf88: Status 404 returned error can't find the container with id a817287dad1cf7fa2edceb42143dfb8692647ee2d1fc151406b27e4a7f72bf88 Nov 02 22:08:36.311243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.311214 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2"] Nov 02 22:08:36.311826 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:36.311798 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90333f30_7fb2_446f_b2b4_a9c2792c5794.slice/crio-1260b1263c4b46b6e53402613eea239ff2a15407c030a32158a40ca890d010ca WatchSource:0}: Error finding container 1260b1263c4b46b6e53402613eea239ff2a15407c030a32158a40ca890d010ca: Status 404 returned error can't find the container with id 1260b1263c4b46b6e53402613eea239ff2a15407c030a32158a40ca890d010ca Nov 02 22:08:36.376295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.376272 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg"] Nov 02 22:08:36.380397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.380379 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:36.390007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.389983 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-361a44\"" Nov 02 22:08:36.393258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.393236 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg"] Nov 02 22:08:36.402292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.402256 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h"] Nov 02 22:08:36.406567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.406548 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:36.410271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.410252 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-361a44-dockercfg-qx2pb\"" Nov 02 22:08:36.413489 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.413461 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h"] Nov 02 22:08:36.490382 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.490353 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4qd\" (UniqueName: \"kubernetes.io/projected/50696516-c79b-4513-99c7-1175999a2e50-kube-api-access-zn4qd\") pod \"payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg\" (UID: \"50696516-c79b-4513-99c7-1175999a2e50\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:36.490550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.490421 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50696516-c79b-4513-99c7-1175999a2e50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg\" (UID: \"50696516-c79b-4513-99c7-1175999a2e50\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:36.490550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.490485 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8x7\" (UniqueName: \"kubernetes.io/projected/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-kube-api-access-6j8x7\") pod \"payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h\" (UID: \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:36.490628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.490578 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h\" (UID: \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:36.592142 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.592106 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8x7\" (UniqueName: \"kubernetes.io/projected/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-kube-api-access-6j8x7\") pod \"payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h\" (UID: \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:36.592302 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.592211 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h\" (UID: \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:36.592372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.592312 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4qd\" (UniqueName: \"kubernetes.io/projected/50696516-c79b-4513-99c7-1175999a2e50-kube-api-access-zn4qd\") pod \"payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg\" (UID: \"50696516-c79b-4513-99c7-1175999a2e50\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:36.592414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.592390 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50696516-c79b-4513-99c7-1175999a2e50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg\" (UID: \"50696516-c79b-4513-99c7-1175999a2e50\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:36.593056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.593029 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h\" (UID: \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:36.593424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.593317 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50696516-c79b-4513-99c7-1175999a2e50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg\" (UID: \"50696516-c79b-4513-99c7-1175999a2e50\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:36.604229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.604203 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4qd\" (UniqueName: \"kubernetes.io/projected/50696516-c79b-4513-99c7-1175999a2e50-kube-api-access-zn4qd\") pod \"payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg\" (UID: \"50696516-c79b-4513-99c7-1175999a2e50\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:36.604387 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.604366 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8x7\" (UniqueName: \"kubernetes.io/projected/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-kube-api-access-6j8x7\") pod \"payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h\" (UID: \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:36.697667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.697592 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:36.723623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.723589 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:36.778273 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.778156 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4"] Nov 02 22:08:36.784773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.784749 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:36.788724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.787506 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4"] Nov 02 22:08:36.797205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.796531 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-310bb3-dockercfg-gqxb2\"" Nov 02 22:08:36.845911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.845878 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-310bb3\"" Nov 02 22:08:36.896707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.896622 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2304b2a9-889a-4265-9ff3-25ac877f2f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4\" (UID: \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:36.896910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.896731 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljfl8\" (UniqueName: \"kubernetes.io/projected/2304b2a9-889a-4265-9ff3-25ac877f2f0d-kube-api-access-ljfl8\") pod \"payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4\" (UID: \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:36.982096 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:36.982061 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50696516_c79b_4513_99c7_1175999a2e50.slice/crio-6040b619de01201b22d37fbeb976d2cd1d86ed22b9ad3c639295e44cea413a99 WatchSource:0}: Error finding container 6040b619de01201b22d37fbeb976d2cd1d86ed22b9ad3c639295e44cea413a99: Status 404 returned error can't find the container with id 6040b619de01201b22d37fbeb976d2cd1d86ed22b9ad3c639295e44cea413a99 Nov 02 22:08:36.983377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.983349 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg"] Nov 02 22:08:36.998542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.998514 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2304b2a9-889a-4265-9ff3-25ac877f2f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4\" (UID: \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:36.998701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.998571 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljfl8\" (UniqueName: \"kubernetes.io/projected/2304b2a9-889a-4265-9ff3-25ac877f2f0d-kube-api-access-ljfl8\") pod \"payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4\" (UID: \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:36.999435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:36.999413 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2304b2a9-889a-4265-9ff3-25ac877f2f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4\" (UID: \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:37.010388 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.010360 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljfl8\" (UniqueName: \"kubernetes.io/projected/2304b2a9-889a-4265-9ff3-25ac877f2f0d-kube-api-access-ljfl8\") pod \"payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4\" (UID: \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:37.016411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.016392 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h"] Nov 02 22:08:37.018831 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:37.018808 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0b2263_76f6_4f5d_b402_be9d5ec3b645.slice/crio-205a5dd3af0b4cfc40559cc7a2cb02f58c3af206b573c732c8c778b55c88c54a WatchSource:0}: Error finding container 205a5dd3af0b4cfc40559cc7a2cb02f58c3af206b573c732c8c778b55c88c54a: Status 404 returned error can't find the container with id 205a5dd3af0b4cfc40559cc7a2cb02f58c3af206b573c732c8c778b55c88c54a Nov 02 22:08:37.102622 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.102591 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:37.146032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.145999 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" event={"ID":"08b0dad1-c312-47fd-ac10-8b4d26a32735","Type":"ContainerStarted","Data":"f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4"} Nov 02 22:08:37.146217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.146049 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" event={"ID":"08b0dad1-c312-47fd-ac10-8b4d26a32735","Type":"ContainerStarted","Data":"a817287dad1cf7fa2edceb42143dfb8692647ee2d1fc151406b27e4a7f72bf88"} Nov 02 22:08:37.146217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.146089 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:37.148103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.148077 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" event={"ID":"50696516-c79b-4513-99c7-1175999a2e50","Type":"ContainerStarted","Data":"958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8"} Nov 02 22:08:37.148233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.148123 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" event={"ID":"50696516-c79b-4513-99c7-1175999a2e50","Type":"ContainerStarted","Data":"6040b619de01201b22d37fbeb976d2cd1d86ed22b9ad3c639295e44cea413a99"} Nov 02 22:08:37.148233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.148163 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:37.150244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.150217 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" event={"ID":"90333f30-7fb2-446f-b2b4-a9c2792c5794","Type":"ContainerStarted","Data":"bc999d2836b4b27856bf0c2d42870cdbce8604109b7fb1fd2747762c838786e1"} Nov 02 22:08:37.150369 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.150265 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" event={"ID":"90333f30-7fb2-446f-b2b4-a9c2792c5794","Type":"ContainerStarted","Data":"1260b1263c4b46b6e53402613eea239ff2a15407c030a32158a40ca890d010ca"} Nov 02 22:08:37.150369 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.150311 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:37.152643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.152621 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" event={"ID":"5b0b2263-76f6-4f5d-b402-be9d5ec3b645","Type":"ContainerStarted","Data":"9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d"} Nov 02 22:08:37.152847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.152832 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:37.152987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.152972 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" event={"ID":"5b0b2263-76f6-4f5d-b402-be9d5ec3b645","Type":"ContainerStarted","Data":"205a5dd3af0b4cfc40559cc7a2cb02f58c3af206b573c732c8c778b55c88c54a"} Nov 02 22:08:37.166123 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.166072 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" podStartSLOduration=2.166055398 podStartE2EDuration="2.166055398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:37.164393244 +0000 UTC m=+645.102506466" watchObservedRunningTime="2026-11-02 22:08:37.166055398 +0000 UTC m=+645.104168616" Nov 02 22:08:37.204834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.204759 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" podStartSLOduration=1.204742904 podStartE2EDuration="1.204742904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:37.203545675 +0000 UTC m=+645.141658894" watchObservedRunningTime="2026-11-02 22:08:37.204742904 +0000 UTC m=+645.142856123" Nov 02 22:08:37.206120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.206081 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" podStartSLOduration=2.206067771 podStartE2EDuration="2.206067771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:37.186888539 +0000 UTC m=+645.125001758" watchObservedRunningTime="2026-11-02 22:08:37.206067771 +0000 UTC m=+645.144180989" Nov 02 22:08:37.221314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.221272 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" podStartSLOduration=1.221263619 podStartE2EDuration="1.221263619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:37.22043529 +0000 UTC m=+645.158548510" watchObservedRunningTime="2026-11-02 22:08:37.221263619 +0000 UTC m=+645.159376836" Nov 02 22:08:37.391048 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.391021 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4"] Nov 02 22:08:37.393431 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:37.393399 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2304b2a9_889a_4265_9ff3_25ac877f2f0d.slice/crio-70b0a68596dd623cb6455d6c9327a2c14994181997863fb272cbf1eea10ed848 WatchSource:0}: Error finding container 70b0a68596dd623cb6455d6c9327a2c14994181997863fb272cbf1eea10ed848: Status 404 returned error can't find the container with id 70b0a68596dd623cb6455d6c9327a2c14994181997863fb272cbf1eea10ed848 Nov 02 22:08:37.580702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.580658 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6"] Nov 02 22:08:37.616920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.616883 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6"] Nov 02 22:08:37.617432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.616997 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:37.620363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.620341 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c302b6\"" Nov 02 22:08:37.642410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.642362 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-c302b6-dockercfg-s9qhl\"" Nov 02 22:08:37.713706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.713664 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccwf\" (UniqueName: \"kubernetes.io/projected/1292bd76-2ee4-43b5-b3fe-f77417b25e30-kube-api-access-6ccwf\") pod \"payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6\" (UID: \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:37.713854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.713814 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1292bd76-2ee4-43b5-b3fe-f77417b25e30-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6\" (UID: \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:37.814889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.814853 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1292bd76-2ee4-43b5-b3fe-f77417b25e30-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6\" (UID: \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:37.815056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.815040 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccwf\" (UniqueName: \"kubernetes.io/projected/1292bd76-2ee4-43b5-b3fe-f77417b25e30-kube-api-access-6ccwf\") pod \"payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6\" (UID: \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:37.817660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.817629 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1292bd76-2ee4-43b5-b3fe-f77417b25e30-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6\" (UID: \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:37.823316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.823290 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccwf\" (UniqueName: \"kubernetes.io/projected/1292bd76-2ee4-43b5-b3fe-f77417b25e30-kube-api-access-6ccwf\") pod \"payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6\" (UID: \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:37.934461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:37.934376 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:38.160537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:38.160500 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" event={"ID":"2304b2a9-889a-4265-9ff3-25ac877f2f0d","Type":"ContainerStarted","Data":"7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7"} Nov 02 22:08:38.160770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:38.160565 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" event={"ID":"2304b2a9-889a-4265-9ff3-25ac877f2f0d","Type":"ContainerStarted","Data":"70b0a68596dd623cb6455d6c9327a2c14994181997863fb272cbf1eea10ed848"} Nov 02 22:08:38.160865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:38.160825 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:38.213897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:38.213346 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" podStartSLOduration=2.213332619 podStartE2EDuration="2.213332619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:38.189486873 +0000 UTC m=+646.127600093" watchObservedRunningTime="2026-11-02 22:08:38.213332619 +0000 UTC m=+646.151445837" Nov 02 22:08:38.214498 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:38.214476 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6"] Nov 02 22:08:38.216761 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:38.216737 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1292bd76_2ee4_43b5_b3fe_f77417b25e30.slice/crio-9cbd22e71b37c16dbbee8e3c0ed5140af8e3f4aa07080d6002148a9014f0f1bd WatchSource:0}: Error finding container 9cbd22e71b37c16dbbee8e3c0ed5140af8e3f4aa07080d6002148a9014f0f1bd: Status 404 returned error can't find the container with id 9cbd22e71b37c16dbbee8e3c0ed5140af8e3f4aa07080d6002148a9014f0f1bd Nov 02 22:08:39.177618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:39.177571 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" event={"ID":"1292bd76-2ee4-43b5-b3fe-f77417b25e30","Type":"ContainerStarted","Data":"34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063"} Nov 02 22:08:39.178062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:39.177644 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" event={"ID":"1292bd76-2ee4-43b5-b3fe-f77417b25e30","Type":"ContainerStarted","Data":"9cbd22e71b37c16dbbee8e3c0ed5140af8e3f4aa07080d6002148a9014f0f1bd"} Nov 02 22:08:39.178062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:39.177814 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:39.210704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:39.195605 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" podStartSLOduration=2.195585448 podStartE2EDuration="2.195585448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:39.193076159 +0000 UTC m=+647.131189378" watchObservedRunningTime="2026-11-02 22:08:39.195585448 +0000 UTC m=+647.133698667" Nov 02 22:08:41.649186 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:08:41.649149 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:08:46.134885 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:46.134846 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:08:47.148912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:47.148865 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:08:47.149433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:47.148950 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:08:47.155829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:47.155798 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:08:48.147883 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.147822 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:08:48.148046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.147921 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:08:48.164852 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.164825 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:08:48.165227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.164908 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:08:48.165772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.165748 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:08:48.826758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.826719 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4"] Nov 02 22:08:48.839752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.839729 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4"] Nov 02 22:08:48.840027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.839978 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.856938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.856652 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a1d78c\"/\"kube-root-ca.crt\"" Nov 02 22:08:48.856938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.856766 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1d78c\"/\"default-dockercfg-sz7zh\"" Nov 02 22:08:48.856938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.856835 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1d78c\"/\"simulator-w6-a1d78c-kserve-self-signed-certs\"" Nov 02 22:08:48.857327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.857273 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a1d78c\"/\"openshift-service-ca.crt\"" Nov 02 22:08:48.865212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.865013 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-dshm\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.865212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.865066 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-model-cache\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.865212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.865107 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/8394a652-8139-4240-b3bb-4ff8b69ea42a-kube-api-access-6tgvw\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.865212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.865160 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-home\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.865212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.865195 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8394a652-8139-4240-b3bb-4ff8b69ea42a-tls-certs\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.865565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.865311 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-tmp-dir\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.966561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.966520 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-tmp-dir\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.966771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.966643 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-dshm\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.966771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.966695 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-model-cache\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.966771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.966739 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/8394a652-8139-4240-b3bb-4ff8b69ea42a-kube-api-access-6tgvw\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.966931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.966868 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-home\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.967000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.966927 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8394a652-8139-4240-b3bb-4ff8b69ea42a-tls-certs\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.967137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.967112 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-model-cache\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.967336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.967313 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-home\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.967550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.967519 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-tmp-dir\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.969627 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.969603 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8394a652-8139-4240-b3bb-4ff8b69ea42a-tls-certs\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.975534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.975507 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-dshm\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:48.977591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:48.977572 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/8394a652-8139-4240-b3bb-4ff8b69ea42a-kube-api-access-6tgvw\") pod \"simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:49.161095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:49.161010 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:49.206708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:49.189034 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:08:49.212016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:49.211228 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:08:49.498391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:49.498361 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4"] Nov 02 22:08:49.499336 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:08:49.499311 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8394a652_8139_4240_b3bb_4ff8b69ea42a.slice/crio-300c741f92ba229575f8be5c30d15bf8464a29d8bdac9e565e890c193093100b WatchSource:0}: Error finding container 300c741f92ba229575f8be5c30d15bf8464a29d8bdac9e565e890c193093100b: Status 404 returned error can't find the container with id 300c741f92ba229575f8be5c30d15bf8464a29d8bdac9e565e890c193093100b Nov 02 22:08:50.215096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.214998 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:08:50.362726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.362662 4309 generic.go:356] "Generic (PLEG): container finished" podID="639a7482-b746-42ea-a91e-95b7a39cd269" containerID="9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d" exitCode=0 Nov 02 22:08:50.362922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.362795 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" event={"ID":"639a7482-b746-42ea-a91e-95b7a39cd269","Type":"ContainerDied","Data":"9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d"} Nov 02 22:08:50.363409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.363381 4309 scope.go:122] "RemoveContainer" containerID="9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d" Nov 02 22:08:50.366001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.365971 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" event={"ID":"8394a652-8139-4240-b3bb-4ff8b69ea42a","Type":"ContainerStarted","Data":"ba56afd4a07a32460fa25e45ec27eca73d5b4abaa7e86147a713cdce1103c9e3"} Nov 02 22:08:50.366137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.366012 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" event={"ID":"8394a652-8139-4240-b3bb-4ff8b69ea42a","Type":"ContainerStarted","Data":"300c741f92ba229575f8be5c30d15bf8464a29d8bdac9e565e890c193093100b"} Nov 02 22:08:50.366765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.366732 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:50.366957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.366939 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:50.367104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.367077 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:50.369860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.369789 4309 generic.go:356] "Generic (PLEG): container finished" podID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerID="ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035" exitCode=0 Nov 02 22:08:50.369927 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.369887 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" event={"ID":"66ff0346-d7be-4890-bdfe-7c7b9d398fd5","Type":"ContainerDied","Data":"ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035"} Nov 02 22:08:50.370232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.370214 4309 scope.go:122] "RemoveContainer" containerID="ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035" Nov 02 22:08:50.372100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.371612 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:50.372100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.371721 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:50.372100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.371768 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:50.481809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:50.481743 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podStartSLOduration=2.481722203 podStartE2EDuration="2.481722203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:50.478947609 +0000 UTC m=+658.417060829" watchObservedRunningTime="2026-11-02 22:08:50.481722203 +0000 UTC m=+658.419835421" Nov 02 22:08:51.176702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:51.175810 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:51.407701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:51.407635 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" event={"ID":"639a7482-b746-42ea-a91e-95b7a39cd269","Type":"ContainerStarted","Data":"6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380"} Nov 02 22:08:51.413845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:51.412264 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" event={"ID":"66ff0346-d7be-4890-bdfe-7c7b9d398fd5","Type":"ContainerStarted","Data":"ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d"} Nov 02 22:08:51.442752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:51.442070 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:51.442752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:51.442192 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:08:51.442752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:51.442237 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 22:08:51.636904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:51.636815 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:51.642818 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:51.642784 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:08:51.643061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:51.643036 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 22:08:52.493706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:52.479811 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:08:52.493706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:52.479873 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 22:08:52.635109 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:52.635074 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:08:52.639054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:52.639004 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 22:08:53.445579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:53.445104 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:08:53.445579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:53.445167 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 22:08:53.636701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:53.636522 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:08:53.636701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:53.636579 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 22:08:54.449560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:54.449091 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:08:54.449560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:54.449159 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 22:08:54.644181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:54.642024 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:08:54.644181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:54.642081 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 22:08:55.442103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:55.442064 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:08:55.442420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:55.442390 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 22:08:55.637741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:55.637285 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:08:55.637741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:55.637344 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 22:08:56.450702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:56.448137 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 22:08:56.450702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:56.448198 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 22:08:56.645314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:56.644899 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 22:08:56.645314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:56.644951 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 22:08:57.246519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.244034 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb"] Nov 02 22:08:57.251353 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.251325 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb"] Nov 02 22:08:57.251520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.251434 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.255952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.255928 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"openshift-service-ca.crt\"" Nov 02 22:08:57.256376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.256356 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"kube-root-ca.crt\"" Nov 02 22:08:57.256583 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.256548 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"simulator-w1-5f4a2f-kserve-self-signed-certs\"" Nov 02 22:08:57.256700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.256621 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"default-dockercfg-7s9mk\"" Nov 02 22:08:57.348826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.348779 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tmp-dir\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.349020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.348837 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-home\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.349020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.348989 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tls-certs\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.349123 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.349102 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-model-cache\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.349224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.349203 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-dshm\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.349299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.349277 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tp58\" (UniqueName: \"kubernetes.io/projected/b6f206e9-742a-416e-b4aa-1bf1a93c301d-kube-api-access-7tp58\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.456196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.456158 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:57.456664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.456412 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-model-cache\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.456664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.456514 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-dshm\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.456664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.456542 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tp58\" (UniqueName: \"kubernetes.io/projected/b6f206e9-742a-416e-b4aa-1bf1a93c301d-kube-api-access-7tp58\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.456664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.456607 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tmp-dir\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.456664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.456633 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-home\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.456948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.456728 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tls-certs\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.457395 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.457370 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:57.457782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.457758 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tmp-dir\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.458247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.458223 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-model-cache\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.458580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.458458 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-home\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.459446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.459426 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:08:57.462581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.462550 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tls-certs\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.468954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.468928 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tp58\" (UniqueName: \"kubernetes.io/projected/b6f206e9-742a-416e-b4aa-1bf1a93c301d-kube-api-access-7tp58\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.472974 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.472928 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-dshm\") pod \"simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.569023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.568786 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:57.777051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.777013 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:57.778458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.778427 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:57.779526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.779500 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:08:57.911276 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:57.911225 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb"] Nov 02 22:08:58.596607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:58.596265 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" event={"ID":"b6f206e9-742a-416e-b4aa-1bf1a93c301d","Type":"ContainerStarted","Data":"4db96f12fa60a28bba7284987236404ca845ba2cebbef73b0c22c4505dba55d7"} Nov 02 22:08:58.596607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:58.596314 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" event={"ID":"b6f206e9-742a-416e-b4aa-1bf1a93c301d","Type":"ContainerStarted","Data":"00b56213cdd8a395d023fa10a1c086a63b0c77b782ac3a9df3193f2a7e7ebabd"} Nov 02 22:08:58.640249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:58.640185 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podStartSLOduration=1.640163365 podStartE2EDuration="1.640163365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:08:58.637022345 +0000 UTC m=+666.575135564" watchObservedRunningTime="2026-11-02 22:08:58.640163365 +0000 UTC m=+666.578276584" Nov 02 22:08:59.162213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.162178 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:59.175320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.175287 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:08:59.569066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.569033 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:08:59.607450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.607416 4309 generic.go:356] "Generic (PLEG): container finished" podID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerID="afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e" exitCode=0 Nov 02 22:08:59.607959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.607768 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" event={"ID":"95a6102b-8f0b-4ecc-8e29-836c57f0420b","Type":"ContainerDied","Data":"afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e"} Nov 02 22:08:59.608211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.608187 4309 scope.go:122] "RemoveContainer" containerID="afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e" Nov 02 22:08:59.610267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.610240 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:59.610390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.610319 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:59.610390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.610369 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:08:59.627732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:08:59.627704 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:09:00.620786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:00.620755 4309 generic.go:356] "Generic (PLEG): container finished" podID="83f302bd-e66a-43b9-b818-8748b284d98f" containerID="1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c" exitCode=0 Nov 02 22:09:00.621345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:00.620865 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" event={"ID":"83f302bd-e66a-43b9-b818-8748b284d98f","Type":"ContainerDied","Data":"1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c"} Nov 02 22:09:00.621345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:00.621231 4309 scope.go:122] "RemoveContainer" containerID="1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c" Nov 02 22:09:00.622713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:00.622627 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:09:00.622825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:00.622763 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:09:00.622825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:00.622813 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:09:00.623339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:00.623318 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" event={"ID":"95a6102b-8f0b-4ecc-8e29-836c57f0420b","Type":"ContainerStarted","Data":"9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee"} Nov 02 22:09:01.103697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:01.100836 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:09:01.360908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:01.360824 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:09:01.362112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:01.362088 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:09:01.362188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:01.362136 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 22:09:01.665060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:01.664775 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" event={"ID":"83f302bd-e66a-43b9-b818-8748b284d98f","Type":"ContainerStarted","Data":"1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257"} Nov 02 22:09:01.804699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:01.804075 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:09:01.813698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:01.806123 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:09:01.813698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:01.806181 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 22:09:02.105697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:02.104792 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:09:02.358465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:02.358383 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:09:02.358465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:02.358436 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 22:09:02.804631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:02.804121 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:09:02.804631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:02.804182 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 22:09:03.358271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:03.358230 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:09:03.358544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:03.358515 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 22:09:03.816761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:03.811218 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:09:03.816761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:03.811273 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 22:09:04.358633 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:04.358598 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:09:04.358848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:04.358659 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 22:09:04.803901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:04.803803 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:09:04.804178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:04.804146 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 22:09:05.358643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:05.358544 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 22:09:05.359127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:05.358624 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 22:09:05.812712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:05.808867 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:09:05.812712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:05.808939 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 22:09:06.548100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:06.548063 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:09:06.552493 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:06.552470 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:09:06.800774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:06.800666 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 22:09:06.800774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:06.800754 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 22:09:07.569099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.569044 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:09:07.582274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.582249 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:09:07.609306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.609279 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb"] Nov 02 22:09:07.617727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.615908 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.620935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.620898 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"openshift-service-ca.crt\"" Nov 02 22:09:07.621285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.621251 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"simulator-w0-310bb3-kserve-self-signed-certs\"" Nov 02 22:09:07.621570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.621541 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"default-dockercfg-cc2pz\"" Nov 02 22:09:07.621903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.621888 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"kube-root-ca.crt\"" Nov 02 22:09:07.624249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.624212 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb"] Nov 02 22:09:07.750815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.750782 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2z8w\" (UniqueName: \"kubernetes.io/projected/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-kube-api-access-m2z8w\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.750989 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.750850 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-model-cache\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.751062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.750978 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tls-certs\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.751120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.751059 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-dshm\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.751120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.751109 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-home\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.751262 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.751236 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tmp-dir\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.771945 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.771922 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:09:07.810010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.809984 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:09:07.811597 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.811569 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:09:07.852721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.852624 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2z8w\" (UniqueName: \"kubernetes.io/projected/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-kube-api-access-m2z8w\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.852892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.852727 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-model-cache\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.852892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.852772 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tls-certs\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.853060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.853039 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-dshm\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.853149 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.853134 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-home\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.853226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.853140 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-model-cache\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.853290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.853208 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tmp-dir\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.853461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.853436 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-home\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.853461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.853450 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tmp-dir\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.855779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.855756 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-dshm\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.856322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.856295 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tls-certs\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.869734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.868929 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2z8w\" (UniqueName: \"kubernetes.io/projected/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-kube-api-access-m2z8w\") pod \"simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:07.936761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:07.936733 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:08.217819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:08.217747 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb"] Nov 02 22:09:08.221054 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:09:08.221026 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd330d8c_bd62_4cb4_a33f_b3604f55bc47.slice/crio-78bcd0b13988d207d13af56e133e49bf1a731a17973e46bb26614b0f264ffb51 WatchSource:0}: Error finding container 78bcd0b13988d207d13af56e133e49bf1a731a17973e46bb26614b0f264ffb51: Status 404 returned error can't find the container with id 78bcd0b13988d207d13af56e133e49bf1a731a17973e46bb26614b0f264ffb51 Nov 02 22:09:08.771170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:08.771133 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" event={"ID":"bd330d8c-bd62-4cb4-a33f-b3604f55bc47","Type":"ContainerStarted","Data":"9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c"} Nov 02 22:09:08.771602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:08.771184 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" event={"ID":"bd330d8c-bd62-4cb4-a33f-b3604f55bc47","Type":"ContainerStarted","Data":"78bcd0b13988d207d13af56e133e49bf1a731a17973e46bb26614b0f264ffb51"} Nov 02 22:09:08.788333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:08.788283 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podStartSLOduration=1.788269465 podStartE2EDuration="1.788269465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:08.787012008 +0000 UTC m=+676.725125226" watchObservedRunningTime="2026-11-02 22:09:08.788269465 +0000 UTC m=+676.726382742" Nov 02 22:09:08.937981 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:08.937937 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:11.596893 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:09:11.596859 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:09:17.937862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:17.937831 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:17.951171 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:17.951145 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:18.843537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:18.843505 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:09:22.921863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:22.921827 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc"] Nov 02 22:09:22.928583 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:22.928559 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:22.932385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:22.932349 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc"] Nov 02 22:09:22.939928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:22.937941 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-719012\"/\"default-dockercfg-q9cxc\"" Nov 02 22:09:22.939928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:22.937971 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-719012\"/\"premium-w3-719012-kserve-self-signed-certs\"" Nov 02 22:09:22.939928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:22.937981 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-719012\"/\"openshift-service-ca.crt\"" Nov 02 22:09:22.939928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:22.938069 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-719012\"/\"kube-root-ca.crt\"" Nov 02 22:09:23.034843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.034806 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0127a148-aec0-4908-a7c5-c8027bc5b451-tls-certs\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.035020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.034973 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-home\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.035020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.035014 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-dshm\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.035108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.035086 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-model-cache\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.035172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.035150 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-tmp-dir\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.035217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.035193 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f2wm\" (UniqueName: \"kubernetes.io/projected/0127a148-aec0-4908-a7c5-c8027bc5b451-kube-api-access-9f2wm\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.136458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.136421 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0127a148-aec0-4908-a7c5-c8027bc5b451-tls-certs\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.136646 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.136538 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-home\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.136646 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.136572 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-dshm\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.136831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.136671 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-model-cache\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.136831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.136742 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-tmp-dir\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.136831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.136773 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f2wm\" (UniqueName: \"kubernetes.io/projected/0127a148-aec0-4908-a7c5-c8027bc5b451-kube-api-access-9f2wm\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.137071 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.137044 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-home\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.137178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.137048 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-model-cache\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.137178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.137135 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-tmp-dir\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.138783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.138757 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-dshm\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.139135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.139115 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0127a148-aec0-4908-a7c5-c8027bc5b451-tls-certs\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.144713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.144653 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f2wm\" (UniqueName: \"kubernetes.io/projected/0127a148-aec0-4908-a7c5-c8027bc5b451-kube-api-access-9f2wm\") pod \"premium-w3-719012-kserve-67cfd8c5cc-97xnc\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.245657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.245564 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:23.525958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.525928 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc"] Nov 02 22:09:23.529411 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:09:23.529372 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0127a148_aec0_4908_a7c5_c8027bc5b451.slice/crio-5229d4785338a4f2bb7f21adb18f41eee21f49825fa6b720d35f4b05ba268058 WatchSource:0}: Error finding container 5229d4785338a4f2bb7f21adb18f41eee21f49825fa6b720d35f4b05ba268058: Status 404 returned error can't find the container with id 5229d4785338a4f2bb7f21adb18f41eee21f49825fa6b720d35f4b05ba268058 Nov 02 22:09:23.863480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.863434 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" event={"ID":"0127a148-aec0-4908-a7c5-c8027bc5b451","Type":"ContainerStarted","Data":"77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d"} Nov 02 22:09:23.863663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.863494 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" event={"ID":"0127a148-aec0-4908-a7c5-c8027bc5b451","Type":"ContainerStarted","Data":"5229d4785338a4f2bb7f21adb18f41eee21f49825fa6b720d35f4b05ba268058"} Nov 02 22:09:23.886791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:23.886734 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podStartSLOduration=1.8867152680000001 podStartE2EDuration="1.886715268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:23.884130087 +0000 UTC m=+691.822243305" watchObservedRunningTime="2026-11-02 22:09:23.886715268 +0000 UTC m=+691.824828487" Nov 02 22:09:24.246621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:24.246531 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:32.989307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:32.989158 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42"] Nov 02 22:09:32.994170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:32.994142 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.007905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.007831 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-361a44\"/\"openshift-service-ca.crt\"" Nov 02 22:09:33.009049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.009026 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-361a44\"/\"kube-root-ca.crt\"" Nov 02 22:09:33.009564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.009285 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-361a44\"/\"premium-w5-361a44-kserve-self-signed-certs\"" Nov 02 22:09:33.011813 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.011788 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-361a44\"/\"default-dockercfg-hxfz5\"" Nov 02 22:09:33.012214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.012195 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42"] Nov 02 22:09:33.055617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.055583 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-tmp-dir\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.055617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.055622 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-home\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.055842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.055642 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-dshm\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.055842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.055791 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ljkj\" (UniqueName: \"kubernetes.io/projected/62a7688c-e69a-4daa-b742-fa4996d2179c-kube-api-access-9ljkj\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.055842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.055813 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7688c-e69a-4daa-b742-fa4996d2179c-tls-certs\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.055937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.055917 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-model-cache\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.157035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.156991 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-model-cache\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.157208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.157160 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-tmp-dir\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.157208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.157193 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-home\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.157282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.157222 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-dshm\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.157328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.157294 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ljkj\" (UniqueName: \"kubernetes.io/projected/62a7688c-e69a-4daa-b742-fa4996d2179c-kube-api-access-9ljkj\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.157381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.157326 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7688c-e69a-4daa-b742-fa4996d2179c-tls-certs\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.157606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.157530 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-model-cache\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.158204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.158173 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-home\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.158497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.158417 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-tmp-dir\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.160283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.160261 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-dshm\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.160656 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.160637 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7688c-e69a-4daa-b742-fa4996d2179c-tls-certs\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.165341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.165315 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ljkj\" (UniqueName: \"kubernetes.io/projected/62a7688c-e69a-4daa-b742-fa4996d2179c-kube-api-access-9ljkj\") pod \"premium-w5-361a44-kserve-5f6bb68996-fhm42\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.246285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.246192 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:33.260089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.260061 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:33.309544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.309498 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:33.592123 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:09:33.592085 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62a7688c_e69a_4daa_b742_fa4996d2179c.slice/crio-4e087cc35f131971db2945919cd42897b64a5b7e2540cefa5701f4f4c2204cae WatchSource:0}: Error finding container 4e087cc35f131971db2945919cd42897b64a5b7e2540cefa5701f4f4c2204cae: Status 404 returned error can't find the container with id 4e087cc35f131971db2945919cd42897b64a5b7e2540cefa5701f4f4c2204cae Nov 02 22:09:33.608516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.608489 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42"] Nov 02 22:09:33.934336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.934246 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" event={"ID":"62a7688c-e69a-4daa-b742-fa4996d2179c","Type":"ContainerStarted","Data":"6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191"} Nov 02 22:09:33.934336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.934289 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" event={"ID":"62a7688c-e69a-4daa-b742-fa4996d2179c","Type":"ContainerStarted","Data":"4e087cc35f131971db2945919cd42897b64a5b7e2540cefa5701f4f4c2204cae"} Nov 02 22:09:33.945815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.945786 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:09:33.949901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:33.949862 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podStartSLOduration=1.949850093 podStartE2EDuration="1.949850093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:09:33.949330205 +0000 UTC m=+701.887443418" watchObservedRunningTime="2026-11-02 22:09:33.949850093 +0000 UTC m=+701.887963311" Nov 02 22:09:34.310579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:34.310551 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:41.595332 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:09:41.595294 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:09:43.310151 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:43.310111 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:43.322824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:43.322794 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:44.009464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:44.009432 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:09:59.483958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.483917 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm"] Nov 02 22:09:59.490517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.490488 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.498465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.498426 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm"] Nov 02 22:09:59.524716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.524666 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-310bb3-openshift-default-dockercfg-qrwjr\"" Nov 02 22:09:59.580326 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.580301 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.580462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.580331 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.580462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.580405 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvmc8\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-kube-api-access-gvmc8\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.580462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.580454 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.580571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.580472 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.580571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.580555 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.580633 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.580605 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.580708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.580695 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.580786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.580767 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.682305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.682275 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.682453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.682356 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.682453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.682374 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.682453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.682399 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvmc8\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-kube-api-access-gvmc8\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.682596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.682548 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.682752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.682731 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.682891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.682874 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.682993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.682972 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.683049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.683020 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.683107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.683081 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.683161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.683149 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.683439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.683403 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.683439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.683403 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.683912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.683892 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.684795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.684768 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.685371 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.685352 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.689569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.689539 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.689939 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.689917 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvmc8\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-kube-api-access-gvmc8\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:09:59.808343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:09:59.808303 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:10:00.088648 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:00.088621 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm"] Nov 02 22:10:00.090519 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:00.090493 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82d2ee2_8555_429a_968b_eb6bb6a1f2c0.slice/crio-bc2f8aa122c6aae3713e9811697326a5545cc32fa22f7fce38c96d003144bfc7 WatchSource:0}: Error finding container bc2f8aa122c6aae3713e9811697326a5545cc32fa22f7fce38c96d003144bfc7: Status 404 returned error can't find the container with id bc2f8aa122c6aae3713e9811697326a5545cc32fa22f7fce38c96d003144bfc7 Nov 02 22:10:00.093139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:00.093104 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:10:00.093236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:00.093170 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:10:00.093236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:00.093198 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:10:00.098841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:00.098819 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" event={"ID":"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0","Type":"ContainerStarted","Data":"bc2f8aa122c6aae3713e9811697326a5545cc32fa22f7fce38c96d003144bfc7"} Nov 02 22:10:01.124613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:01.124571 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" event={"ID":"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0","Type":"ContainerStarted","Data":"a0ac5e26ec29ea4c810fc2cdc05cc281bd3981da2ddc9e95fbaccf6a662b5c45"} Nov 02 22:10:01.147311 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:01.147261 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" podStartSLOduration=2.1472484 podStartE2EDuration="2.1472484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:01.145073971 +0000 UTC m=+729.083187189" watchObservedRunningTime="2026-11-02 22:10:01.1472484 +0000 UTC m=+729.085361617" Nov 02 22:10:01.808961 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:01.808918 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:10:01.810932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:01.810891 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:10:01.811075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:01.810959 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:10:02.810056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:02.810006 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:10:02.810527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:02.810063 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:10:03.811024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:03.810267 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:10:03.811024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:03.810329 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:10:04.810238 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:04.810191 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:10:04.810565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:04.810259 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:10:05.811140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:05.811098 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 22:10:05.811636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:05.811348 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 22:10:06.856846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:06.856809 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:10:06.857535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:06.857509 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:10:06.859101 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:06.859081 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:10:11.598366 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:10:11.598324 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:10:14.093867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.093835 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px"] Nov 02 22:10:14.098232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.098211 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.105251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.105224 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2532af-openshift-default-dockercfg-9p29v\"" Nov 02 22:10:14.113471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.113444 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px"] Nov 02 22:10:14.173515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.173484 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.173707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.173555 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.173707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.173651 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.173801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.173720 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b72vk\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-kube-api-access-b72vk\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.173847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.173812 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.173905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.173886 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.173960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.173918 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.174044 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.174027 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.174104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.174060 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275104 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275167 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275202 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b72vk\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-kube-api-access-b72vk\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275233 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275290 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275313 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275359 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275389 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275515 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275593 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275770 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.275996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.275792 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.276105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.276007 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.276365 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.276342 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.277787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.277767 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.278471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.278455 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.282692 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.282660 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.282948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.282918 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b72vk\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-kube-api-access-b72vk\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-nc2px\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.415205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.414875 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:14.763328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.763288 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px"] Nov 02 22:10:14.765121 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:14.765096 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8715498_8ba4_45f4_8214_fe3bd5c89fcb.slice/crio-6565d6b2dc7a7f7b74655fe5050e2222a595003c0bbba11da26b64c97ffb6851 WatchSource:0}: Error finding container 6565d6b2dc7a7f7b74655fe5050e2222a595003c0bbba11da26b64c97ffb6851: Status 404 returned error can't find the container with id 6565d6b2dc7a7f7b74655fe5050e2222a595003c0bbba11da26b64c97ffb6851 Nov 02 22:10:14.767559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.767527 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:10:14.767644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.767593 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:10:14.767644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:14.767622 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:10:15.020761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.020658 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw"] Nov 02 22:10:15.026139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.026085 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.029789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.029753 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw"] Nov 02 22:10:15.029914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.029829 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:10:15.029914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.029891 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:10:15.029914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.029903 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:10:15.030951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.030928 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:10:15.192495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.192463 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd2315f9-lhqxw\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.192975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.192521 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd2315f9-lhqxw\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.192975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.192651 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4b7\" (UniqueName: \"kubernetes.io/projected/d5e9142f-cfc2-4753-8f05-677b8c627623-kube-api-access-nh4b7\") pod \"maas-api-revoke-sub-bd2315f9-lhqxw\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.233533 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.233493 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" event={"ID":"a8715498-8ba4-45f4-8214-fe3bd5c89fcb","Type":"ContainerStarted","Data":"eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98"} Nov 02 22:10:15.233889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.233566 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" event={"ID":"a8715498-8ba4-45f4-8214-fe3bd5c89fcb","Type":"ContainerStarted","Data":"6565d6b2dc7a7f7b74655fe5050e2222a595003c0bbba11da26b64c97ffb6851"} Nov 02 22:10:15.255020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.254961 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" podStartSLOduration=1.254941429 podStartE2EDuration="1.254941429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:15.254727637 +0000 UTC m=+743.192840858" watchObservedRunningTime="2026-11-02 22:10:15.254941429 +0000 UTC m=+743.193054648" Nov 02 22:10:15.297746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.294425 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd2315f9-lhqxw\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.297746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.294508 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd2315f9-lhqxw\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.297746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.294557 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4b7\" (UniqueName: \"kubernetes.io/projected/d5e9142f-cfc2-4753-8f05-677b8c627623-kube-api-access-nh4b7\") pod \"maas-api-revoke-sub-bd2315f9-lhqxw\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.297746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.295796 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd2315f9-lhqxw\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.301352 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.301318 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd2315f9-lhqxw\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.313612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.313577 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4b7\" (UniqueName: \"kubernetes.io/projected/d5e9142f-cfc2-4753-8f05-677b8c627623-kube-api-access-nh4b7\") pod \"maas-api-revoke-sub-bd2315f9-lhqxw\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.344421 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.344387 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:15.415962 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.415927 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:15.417374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.417343 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 22:10:15.417534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.417396 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 22:10:15.651807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.651773 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw"] Nov 02 22:10:15.653390 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:15.653356 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e9142f_cfc2_4753_8f05_677b8c627623.slice/crio-8d6d4dd8bbfac7da59076e318824a43400f7e40885566785be2f75d1ad2ffea4 WatchSource:0}: Error finding container 8d6d4dd8bbfac7da59076e318824a43400f7e40885566785be2f75d1ad2ffea4: Status 404 returned error can't find the container with id 8d6d4dd8bbfac7da59076e318824a43400f7e40885566785be2f75d1ad2ffea4 Nov 02 22:10:15.655659 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:15.655638 4309 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:10:16.242797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:16.242760 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" event={"ID":"d5e9142f-cfc2-4753-8f05-677b8c627623","Type":"ContainerStarted","Data":"8d6d4dd8bbfac7da59076e318824a43400f7e40885566785be2f75d1ad2ffea4"} Nov 02 22:10:16.416163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:16.416122 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 22:10:16.416370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:16.416192 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 22:10:17.426696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:17.416102 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 22:10:17.426696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:17.416172 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 22:10:18.416242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:18.416206 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 22:10:18.416417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:18.416269 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 22:10:18.838698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:18.828954 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9"] Nov 02 22:10:18.853751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:18.849865 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9"] Nov 02 22:10:18.853751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:18.849989 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:18.957221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:18.957180 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51fe0b51-sgjl9\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:18.957527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:18.957499 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92gxv\" (UniqueName: \"kubernetes.io/projected/47ae7159-95e3-478f-b026-3e0335f441a5-kube-api-access-92gxv\") pod \"maas-api-revoke-sub-51fe0b51-sgjl9\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:18.957773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:18.957752 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51fe0b51-sgjl9\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:19.058966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.058930 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51fe0b51-sgjl9\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:19.059157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.059077 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51fe0b51-sgjl9\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:19.059222 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.059159 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92gxv\" (UniqueName: \"kubernetes.io/projected/47ae7159-95e3-478f-b026-3e0335f441a5-kube-api-access-92gxv\") pod \"maas-api-revoke-sub-51fe0b51-sgjl9\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:19.060293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.060260 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51fe0b51-sgjl9\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:19.062178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.062155 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51fe0b51-sgjl9\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:19.067503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.067479 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92gxv\" (UniqueName: \"kubernetes.io/projected/47ae7159-95e3-478f-b026-3e0335f441a5-kube-api-access-92gxv\") pod \"maas-api-revoke-sub-51fe0b51-sgjl9\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:19.127847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.127758 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp"] Nov 02 22:10:19.134340 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.134311 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.139866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.139836 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp"] Nov 02 22:10:19.243873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.243836 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:19.262491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.262457 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-197d689b-hnlqp\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.262618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.262507 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-197d689b-hnlqp\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.262707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.262657 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dx6\" (UniqueName: \"kubernetes.io/projected/eabdc1e2-b48e-4979-93de-4589e5c033fd-kube-api-access-w6dx6\") pod \"maas-api-revoke-sub-197d689b-hnlqp\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.368468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.363708 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-197d689b-hnlqp\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.368468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.363772 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-197d689b-hnlqp\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.368468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.363923 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dx6\" (UniqueName: \"kubernetes.io/projected/eabdc1e2-b48e-4979-93de-4589e5c033fd-kube-api-access-w6dx6\") pod \"maas-api-revoke-sub-197d689b-hnlqp\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.368468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.365206 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-197d689b-hnlqp\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.373162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.373102 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dx6\" (UniqueName: \"kubernetes.io/projected/eabdc1e2-b48e-4979-93de-4589e5c033fd-kube-api-access-w6dx6\") pod \"maas-api-revoke-sub-197d689b-hnlqp\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.400404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.399363 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-197d689b-hnlqp\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.420497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.420057 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 22:10:19.420497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.420113 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 22:10:19.457115 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.456848 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:19.556928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.556148 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs"] Nov 02 22:10:19.593030 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.566117 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9"] Nov 02 22:10:19.593030 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.566229 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.593030 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:19.578347 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ae7159_95e3_478f_b026_3e0335f441a5.slice/crio-5873397b05bfeabfcdb57fdd0d9c063f6face60d6fb95b4d219c0478663226f4 WatchSource:0}: Error finding container 5873397b05bfeabfcdb57fdd0d9c063f6face60d6fb95b4d219c0478663226f4: Status 404 returned error can't find the container with id 5873397b05bfeabfcdb57fdd0d9c063f6face60d6fb95b4d219c0478663226f4 Nov 02 22:10:19.639886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.639857 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs"] Nov 02 22:10:19.669391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.669265 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20cded6-8tfvs\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.669391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.669370 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvksc\" (UniqueName: \"kubernetes.io/projected/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-kube-api-access-cvksc\") pod \"maas-api-revoke-sub-d20cded6-8tfvs\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.669599 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.669478 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20cded6-8tfvs\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.770693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.770637 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20cded6-8tfvs\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.770876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.770775 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20cded6-8tfvs\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.770876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.770800 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvksc\" (UniqueName: \"kubernetes.io/projected/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-kube-api-access-cvksc\") pod \"maas-api-revoke-sub-d20cded6-8tfvs\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.771447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.771423 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20cded6-8tfvs\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.773047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.773016 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20cded6-8tfvs\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.803699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.781878 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp"] Nov 02 22:10:19.803699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.799369 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvksc\" (UniqueName: \"kubernetes.io/projected/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-kube-api-access-cvksc\") pod \"maas-api-revoke-sub-d20cded6-8tfvs\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:19.822685 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:19.822642 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabdc1e2_b48e_4979_93de_4589e5c033fd.slice/crio-06fd5170e5dc59532dc82ad3b661679fe36574c82a962ca1f430e970e9823559 WatchSource:0}: Error finding container 06fd5170e5dc59532dc82ad3b661679fe36574c82a962ca1f430e970e9823559: Status 404 returned error can't find the container with id 06fd5170e5dc59532dc82ad3b661679fe36574c82a962ca1f430e970e9823559 Nov 02 22:10:19.943044 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:19.943021 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:20.127087 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.123638 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8"] Nov 02 22:10:20.130502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.130431 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.130900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.130880 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8"] Nov 02 22:10:20.176967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.176898 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx4t2\" (UniqueName: \"kubernetes.io/projected/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-kube-api-access-cx4t2\") pod \"maas-api-revoke-sub-61a2b4f4-fmdc8\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.177111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.177013 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a2b4f4-fmdc8\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.177111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.177047 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a2b4f4-fmdc8\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.257354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.257325 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs"] Nov 02 22:10:20.258802 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:20.258776 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e73dbf9_dc28_4828_ba5f_08acf8bae7da.slice/crio-1817e95ce9ea0ad052c72b1745a8d0e60158a31295d2d1593480a4d4f12b8ab8 WatchSource:0}: Error finding container 1817e95ce9ea0ad052c72b1745a8d0e60158a31295d2d1593480a4d4f12b8ab8: Status 404 returned error can't find the container with id 1817e95ce9ea0ad052c72b1745a8d0e60158a31295d2d1593480a4d4f12b8ab8 Nov 02 22:10:20.272670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.272638 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" event={"ID":"5e73dbf9-dc28-4828-ba5f-08acf8bae7da","Type":"ContainerStarted","Data":"1817e95ce9ea0ad052c72b1745a8d0e60158a31295d2d1593480a4d4f12b8ab8"} Nov 02 22:10:20.273967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.273940 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" event={"ID":"eabdc1e2-b48e-4979-93de-4589e5c033fd","Type":"ContainerStarted","Data":"06fd5170e5dc59532dc82ad3b661679fe36574c82a962ca1f430e970e9823559"} Nov 02 22:10:20.275336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.275305 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" event={"ID":"47ae7159-95e3-478f-b026-3e0335f441a5","Type":"ContainerStarted","Data":"5873397b05bfeabfcdb57fdd0d9c063f6face60d6fb95b4d219c0478663226f4"} Nov 02 22:10:20.278064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.278041 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx4t2\" (UniqueName: \"kubernetes.io/projected/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-kube-api-access-cx4t2\") pod \"maas-api-revoke-sub-61a2b4f4-fmdc8\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.278157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.278141 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a2b4f4-fmdc8\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.278213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.278199 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a2b4f4-fmdc8\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.278877 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.278856 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a2b4f4-fmdc8\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.280769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.280747 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a2b4f4-fmdc8\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.323168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.323135 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx4t2\" (UniqueName: \"kubernetes.io/projected/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-kube-api-access-cx4t2\") pod \"maas-api-revoke-sub-61a2b4f4-fmdc8\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.416736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.416695 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 02 22:10:20.416910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.416757 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 02 22:10:20.458858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.458761 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:20.850566 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:20.850535 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f3aa9b_f01f_49c5_87f8_4d3e5ba1a906.slice/crio-0c1a99b2edbafeada1ba0c62c4ac9f24f979694ada91b70613b30961ac856d0e WatchSource:0}: Error finding container 0c1a99b2edbafeada1ba0c62c4ac9f24f979694ada91b70613b30961ac856d0e: Status 404 returned error can't find the container with id 0c1a99b2edbafeada1ba0c62c4ac9f24f979694ada91b70613b30961ac856d0e Nov 02 22:10:20.851219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:20.851191 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8"] Nov 02 22:10:21.331946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.331904 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" event={"ID":"47ae7159-95e3-478f-b026-3e0335f441a5","Type":"ContainerStarted","Data":"c302d49329a36f2a390b53106ab1c5ef957192ead0577476df7e72e3e95102c2"} Nov 02 22:10:21.336850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.336821 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" event={"ID":"d5e9142f-cfc2-4753-8f05-677b8c627623","Type":"ContainerDied","Data":"34064baa5e5057b5a68b35d1a1461ac0824332cc566fe5d557c585623076b08e"} Nov 02 22:10:21.337140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.335719 4309 generic.go:356] "Generic (PLEG): container finished" podID="d5e9142f-cfc2-4753-8f05-677b8c627623" containerID="34064baa5e5057b5a68b35d1a1461ac0824332cc566fe5d557c585623076b08e" exitCode=0 Nov 02 22:10:21.340199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.339410 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" event={"ID":"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906","Type":"ContainerStarted","Data":"0c1a99b2edbafeada1ba0c62c4ac9f24f979694ada91b70613b30961ac856d0e"} Nov 02 22:10:21.342398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.342355 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" event={"ID":"5e73dbf9-dc28-4828-ba5f-08acf8bae7da","Type":"ContainerStarted","Data":"e786af591d6e337b84095e660f3d791b761b3d457ac4e56d3e78600e5e8bcae8"} Nov 02 22:10:21.350645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.350588 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" podStartSLOduration=2.303226152 podStartE2EDuration="3.350571155s" totalImagesPullingTime="1.047345003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:21.347147063 +0000 UTC m=+749.285260283" watchObservedRunningTime="2026-11-02 22:10:21.350571155 +0000 UTC m=+749.288684385" Nov 02 22:10:21.367621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.367563 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" podStartSLOduration=1.3675445210000001 podStartE2EDuration="1.367544521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:21.366695826 +0000 UTC m=+749.304809044" watchObservedRunningTime="2026-11-02 22:10:21.367544521 +0000 UTC m=+749.305657741" Nov 02 22:10:21.408120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.408063 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" podStartSLOduration=2.408044791 podStartE2EDuration="2.408044791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:21.405640587 +0000 UTC m=+749.343753805" watchObservedRunningTime="2026-11-02 22:10:21.408044791 +0000 UTC m=+749.346158013" Nov 02 22:10:21.620536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.617582 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:21.620536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.618983 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:21.625940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:21.625916 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:10:22.350532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.350498 4309 generic.go:356] "Generic (PLEG): container finished" podID="5e73dbf9-dc28-4828-ba5f-08acf8bae7da" containerID="e786af591d6e337b84095e660f3d791b761b3d457ac4e56d3e78600e5e8bcae8" exitCode=0 Nov 02 22:10:22.351051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.350581 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" event={"ID":"5e73dbf9-dc28-4828-ba5f-08acf8bae7da","Type":"ContainerDied","Data":"e786af591d6e337b84095e660f3d791b761b3d457ac4e56d3e78600e5e8bcae8"} Nov 02 22:10:22.352357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.352328 4309 generic.go:356] "Generic (PLEG): container finished" podID="eabdc1e2-b48e-4979-93de-4589e5c033fd" containerID="6463e53f82e58cc3078bc53c79519e1b3d0787bd92fbb5b564f99997449218b9" exitCode=0 Nov 02 22:10:22.352496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.352404 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" event={"ID":"eabdc1e2-b48e-4979-93de-4589e5c033fd","Type":"ContainerDied","Data":"6463e53f82e58cc3078bc53c79519e1b3d0787bd92fbb5b564f99997449218b9"} Nov 02 22:10:22.354082 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.354062 4309 generic.go:356] "Generic (PLEG): container finished" podID="47ae7159-95e3-478f-b026-3e0335f441a5" containerID="c302d49329a36f2a390b53106ab1c5ef957192ead0577476df7e72e3e95102c2" exitCode=0 Nov 02 22:10:22.354200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.354093 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" event={"ID":"47ae7159-95e3-478f-b026-3e0335f441a5","Type":"ContainerDied","Data":"c302d49329a36f2a390b53106ab1c5ef957192ead0577476df7e72e3e95102c2"} Nov 02 22:10:22.355823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.355802 4309 generic.go:356] "Generic (PLEG): container finished" podID="73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906" containerID="bc0a6dbce3fab541c35fce2c88ea8f27746d1c5a7bb69b1a57f0b8b186ecbddf" exitCode=0 Nov 02 22:10:22.355941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.355861 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" event={"ID":"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906","Type":"ContainerDied","Data":"bc0a6dbce3fab541c35fce2c88ea8f27746d1c5a7bb69b1a57f0b8b186ecbddf"} Nov 02 22:10:22.520635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.520609 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:22.630741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.630642 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e9142f-cfc2-4753-8f05-677b8c627623-kube-api-access-nh4b7\" (UniqueName: \"kubernetes.io/projected/d5e9142f-cfc2-4753-8f05-677b8c627623-kube-api-access-nh4b7\") pod \"d5e9142f-cfc2-4753-8f05-677b8c627623\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " Nov 02 22:10:22.630902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.630746 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-openshift-ca\") pod \"d5e9142f-cfc2-4753-8f05-677b8c627623\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " Nov 02 22:10:22.630902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.630864 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-xks-ca\") pod \"d5e9142f-cfc2-4753-8f05-677b8c627623\" (UID: \"d5e9142f-cfc2-4753-8f05-677b8c627623\") " Nov 02 22:10:22.631084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.631051 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-openshift-ca" pod "d5e9142f-cfc2-4753-8f05-677b8c627623" (UID: "d5e9142f-cfc2-4753-8f05-677b8c627623"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:22.631366 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.631348 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:22.632954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.632916 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-xks-ca" pod "d5e9142f-cfc2-4753-8f05-677b8c627623" (UID: "d5e9142f-cfc2-4753-8f05-677b8c627623"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:22.633077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.633004 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e9142f-cfc2-4753-8f05-677b8c627623-kube-api-access-nh4b7" pod "d5e9142f-cfc2-4753-8f05-677b8c627623" (UID: "d5e9142f-cfc2-4753-8f05-677b8c627623"). InnerVolumeSpecName "kube-api-access-nh4b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:22.732342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.732304 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh4b7\" (UniqueName: \"kubernetes.io/projected/d5e9142f-cfc2-4753-8f05-677b8c627623-kube-api-access-nh4b7\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:22.732342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:22.732339 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e9142f-cfc2-4753-8f05-677b8c627623-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.364282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.364211 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" event={"ID":"d5e9142f-cfc2-4753-8f05-677b8c627623","Type":"ContainerDied","Data":"8d6d4dd8bbfac7da59076e318824a43400f7e40885566785be2f75d1ad2ffea4"} Nov 02 22:10:23.364282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.364282 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw" Nov 02 22:10:23.364813 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.364306 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d6d4dd8bbfac7da59076e318824a43400f7e40885566785be2f75d1ad2ffea4" Nov 02 22:10:23.577079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.576478 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:23.650133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.647733 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-kube-api-access-cvksc\" (UniqueName: \"kubernetes.io/projected/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-kube-api-access-cvksc\") pod \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " Nov 02 22:10:23.650133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.647882 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-openshift-ca\") pod \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " Nov 02 22:10:23.650133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.647991 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-xks-ca\") pod \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\" (UID: \"5e73dbf9-dc28-4828-ba5f-08acf8bae7da\") " Nov 02 22:10:23.650133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.649420 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-openshift-ca" pod "5e73dbf9-dc28-4828-ba5f-08acf8bae7da" (UID: "5e73dbf9-dc28-4828-ba5f-08acf8bae7da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:23.653758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.653719 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-kube-api-access-cvksc" pod "5e73dbf9-dc28-4828-ba5f-08acf8bae7da" (UID: "5e73dbf9-dc28-4828-ba5f-08acf8bae7da"). InnerVolumeSpecName "kube-api-access-cvksc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:23.653952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.653925 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-xks-ca" pod "5e73dbf9-dc28-4828-ba5f-08acf8bae7da" (UID: "5e73dbf9-dc28-4828-ba5f-08acf8bae7da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:23.654204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.654182 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:23.657833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.657808 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:23.680500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.680478 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:23.748714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.748664 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-openshift-ca\") pod \"eabdc1e2-b48e-4979-93de-4589e5c033fd\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " Nov 02 22:10:23.748873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.748803 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-kube-api-access-cx4t2\" (UniqueName: \"kubernetes.io/projected/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-kube-api-access-cx4t2\") pod \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " Nov 02 22:10:23.748873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.748840 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-xks-ca\") pod \"eabdc1e2-b48e-4979-93de-4589e5c033fd\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " Nov 02 22:10:23.748957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.748941 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ae7159-95e3-478f-b026-3e0335f441a5-kube-api-access-92gxv\" (UniqueName: \"kubernetes.io/projected/47ae7159-95e3-478f-b026-3e0335f441a5-kube-api-access-92gxv\") pod \"47ae7159-95e3-478f-b026-3e0335f441a5\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " Nov 02 22:10:23.749007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.748993 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-xks-ca\") pod \"47ae7159-95e3-478f-b026-3e0335f441a5\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " Nov 02 22:10:23.749068 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749054 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-openshift-ca\") pod \"47ae7159-95e3-478f-b026-3e0335f441a5\" (UID: \"47ae7159-95e3-478f-b026-3e0335f441a5\") " Nov 02 22:10:23.749117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749085 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabdc1e2-b48e-4979-93de-4589e5c033fd-kube-api-access-w6dx6\" (UniqueName: \"kubernetes.io/projected/eabdc1e2-b48e-4979-93de-4589e5c033fd-kube-api-access-w6dx6\") pod \"eabdc1e2-b48e-4979-93de-4589e5c033fd\" (UID: \"eabdc1e2-b48e-4979-93de-4589e5c033fd\") " Nov 02 22:10:23.749117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749086 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-openshift-ca" pod "eabdc1e2-b48e-4979-93de-4589e5c033fd" (UID: "eabdc1e2-b48e-4979-93de-4589e5c033fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:23.749230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749131 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-xks-ca\") pod \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " Nov 02 22:10:23.749230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749158 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-openshift-ca\") pod \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\" (UID: \"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906\") " Nov 02 22:10:23.749450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749429 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-openshift-ca" pod "47ae7159-95e3-478f-b026-3e0335f441a5" (UID: "47ae7159-95e3-478f-b026-3e0335f441a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:23.749790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749764 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-openshift-ca" pod "73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906" (UID: "73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:23.749884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749809 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.749884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749823 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.749884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749833 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvksc\" (UniqueName: \"kubernetes.io/projected/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-kube-api-access-cvksc\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.749884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749842 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.749884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.749853 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e73dbf9-dc28-4828-ba5f-08acf8bae7da-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.751596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.751555 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-xks-ca" pod "eabdc1e2-b48e-4979-93de-4589e5c033fd" (UID: "eabdc1e2-b48e-4979-93de-4589e5c033fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:23.751596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.751571 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-xks-ca" pod "47ae7159-95e3-478f-b026-3e0335f441a5" (UID: "47ae7159-95e3-478f-b026-3e0335f441a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:23.751596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.751587 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ae7159-95e3-478f-b026-3e0335f441a5-kube-api-access-92gxv" pod "47ae7159-95e3-478f-b026-3e0335f441a5" (UID: "47ae7159-95e3-478f-b026-3e0335f441a5"). InnerVolumeSpecName "kube-api-access-92gxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:23.751852 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.751731 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-kube-api-access-cx4t2" pod "73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906" (UID: "73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906"). InnerVolumeSpecName "kube-api-access-cx4t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:23.752079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.752056 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-xks-ca" pod "73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906" (UID: "73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:23.752079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.752067 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabdc1e2-b48e-4979-93de-4589e5c033fd-kube-api-access-w6dx6" pod "eabdc1e2-b48e-4979-93de-4589e5c033fd" (UID: "eabdc1e2-b48e-4979-93de-4589e5c033fd"). InnerVolumeSpecName "kube-api-access-w6dx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:23.851495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.851457 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.851495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.851493 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.851762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.851509 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx4t2\" (UniqueName: \"kubernetes.io/projected/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906-kube-api-access-cx4t2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.851762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.851524 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabdc1e2-b48e-4979-93de-4589e5c033fd-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.851762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.851539 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92gxv\" (UniqueName: \"kubernetes.io/projected/47ae7159-95e3-478f-b026-3e0335f441a5-kube-api-access-92gxv\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.851762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.851554 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47ae7159-95e3-478f-b026-3e0335f441a5-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:23.851762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:23.851569 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dx6\" (UniqueName: \"kubernetes.io/projected/eabdc1e2-b48e-4979-93de-4589e5c033fd-kube-api-access-w6dx6\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:24.373252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.373221 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" Nov 02 22:10:24.373754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.373248 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs" event={"ID":"5e73dbf9-dc28-4828-ba5f-08acf8bae7da","Type":"ContainerDied","Data":"1817e95ce9ea0ad052c72b1745a8d0e60158a31295d2d1593480a4d4f12b8ab8"} Nov 02 22:10:24.373754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.373305 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1817e95ce9ea0ad052c72b1745a8d0e60158a31295d2d1593480a4d4f12b8ab8" Nov 02 22:10:24.374765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.374742 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" Nov 02 22:10:24.374902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.374742 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp" event={"ID":"eabdc1e2-b48e-4979-93de-4589e5c033fd","Type":"ContainerDied","Data":"06fd5170e5dc59532dc82ad3b661679fe36574c82a962ca1f430e970e9823559"} Nov 02 22:10:24.374902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.374877 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06fd5170e5dc59532dc82ad3b661679fe36574c82a962ca1f430e970e9823559" Nov 02 22:10:24.376289 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.376270 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" Nov 02 22:10:24.376289 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.376281 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9" event={"ID":"47ae7159-95e3-478f-b026-3e0335f441a5","Type":"ContainerDied","Data":"5873397b05bfeabfcdb57fdd0d9c063f6face60d6fb95b4d219c0478663226f4"} Nov 02 22:10:24.376450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.376315 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5873397b05bfeabfcdb57fdd0d9c063f6face60d6fb95b4d219c0478663226f4" Nov 02 22:10:24.377897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.377878 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" event={"ID":"73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906","Type":"ContainerDied","Data":"0c1a99b2edbafeada1ba0c62c4ac9f24f979694ada91b70613b30961ac856d0e"} Nov 02 22:10:24.378015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.377916 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c1a99b2edbafeada1ba0c62c4ac9f24f979694ada91b70613b30961ac856d0e" Nov 02 22:10:24.378015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:24.377917 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8" Nov 02 22:10:30.415168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415138 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h"] Nov 02 22:10:30.415766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415750 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d5e9142f-cfc2-4753-8f05-677b8c627623" containerName="revoke-keys" Nov 02 22:10:30.415856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415767 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e9142f-cfc2-4753-8f05-677b8c627623" containerName="revoke-keys" Nov 02 22:10:30.415856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415782 4309 cpu_manager.go:414] "containerMap: removing container" podUID="eabdc1e2-b48e-4979-93de-4589e5c033fd" containerName="revoke-keys" Nov 02 22:10:30.415856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415787 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabdc1e2-b48e-4979-93de-4589e5c033fd" containerName="revoke-keys" Nov 02 22:10:30.415856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415797 4309 cpu_manager.go:414] "containerMap: removing container" podUID="5e73dbf9-dc28-4828-ba5f-08acf8bae7da" containerName="revoke-keys" Nov 02 22:10:30.415856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415801 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e73dbf9-dc28-4828-ba5f-08acf8bae7da" containerName="revoke-keys" Nov 02 22:10:30.415856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415822 4309 cpu_manager.go:414] "containerMap: removing container" podUID="47ae7159-95e3-478f-b026-3e0335f441a5" containerName="revoke-keys" Nov 02 22:10:30.415856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415827 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ae7159-95e3-478f-b026-3e0335f441a5" containerName="revoke-keys" Nov 02 22:10:30.415856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415838 4309 cpu_manager.go:414] "containerMap: removing container" podUID="73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906" containerName="revoke-keys" Nov 02 22:10:30.415856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415843 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906" containerName="revoke-keys" Nov 02 22:10:30.416111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415943 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e9142f-cfc2-4753-8f05-677b8c627623" containerName="revoke-keys" Nov 02 22:10:30.416111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415953 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabdc1e2-b48e-4979-93de-4589e5c033fd" containerName="revoke-keys" Nov 02 22:10:30.416111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415959 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e73dbf9-dc28-4828-ba5f-08acf8bae7da" containerName="revoke-keys" Nov 02 22:10:30.416111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415967 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ae7159-95e3-478f-b026-3e0335f441a5" containerName="revoke-keys" Nov 02 22:10:30.416111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.415973 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906" containerName="revoke-keys" Nov 02 22:10:30.420567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.420551 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.425353 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.425323 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:10:30.425834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.425809 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:10:30.425964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.425866 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:10:30.425964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.425870 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:10:30.426289 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.426269 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h"] Nov 02 22:10:30.530522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.530488 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91259f86-kxq5h\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.530723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.530653 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srltm\" (UniqueName: \"kubernetes.io/projected/135d60c1-4a7f-410d-962e-fe251e968897-kube-api-access-srltm\") pod \"maas-api-revoke-sub-91259f86-kxq5h\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.530723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.530727 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91259f86-kxq5h\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.632388 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.632347 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91259f86-kxq5h\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.632552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.632470 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srltm\" (UniqueName: \"kubernetes.io/projected/135d60c1-4a7f-410d-962e-fe251e968897-kube-api-access-srltm\") pod \"maas-api-revoke-sub-91259f86-kxq5h\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.632552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.632512 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91259f86-kxq5h\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.633009 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.632986 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91259f86-kxq5h\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.634998 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.634970 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91259f86-kxq5h\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.641188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.641161 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srltm\" (UniqueName: \"kubernetes.io/projected/135d60c1-4a7f-410d-962e-fe251e968897-kube-api-access-srltm\") pod \"maas-api-revoke-sub-91259f86-kxq5h\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.711165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.711088 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm"] Nov 02 22:10:30.715358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.715338 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:30.722280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.722253 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm"] Nov 02 22:10:30.737777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.737747 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:30.834993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.834958 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe734ffb-nrnqm\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:30.835157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.835114 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe734ffb-nrnqm\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:30.835383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.835357 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ssp\" (UniqueName: \"kubernetes.io/projected/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-kube-api-access-c2ssp\") pod \"maas-api-revoke-sub-fe734ffb-nrnqm\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:30.936032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.935998 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ssp\" (UniqueName: \"kubernetes.io/projected/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-kube-api-access-c2ssp\") pod \"maas-api-revoke-sub-fe734ffb-nrnqm\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:30.936032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.936040 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe734ffb-nrnqm\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:30.936251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.936158 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe734ffb-nrnqm\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:30.937010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.936967 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe734ffb-nrnqm\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:30.938687 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.938653 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe734ffb-nrnqm\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:30.943424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:30.943396 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ssp\" (UniqueName: \"kubernetes.io/projected/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-kube-api-access-c2ssp\") pod \"maas-api-revoke-sub-fe734ffb-nrnqm\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:31.021461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:31.021433 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h"] Nov 02 22:10:31.022751 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:31.022725 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135d60c1_4a7f_410d_962e_fe251e968897.slice/crio-46d92404762ccc1bfdd53a5c4802ec0e0271608c87f618d86eb9c5f79046b277 WatchSource:0}: Error finding container 46d92404762ccc1bfdd53a5c4802ec0e0271608c87f618d86eb9c5f79046b277: Status 404 returned error can't find the container with id 46d92404762ccc1bfdd53a5c4802ec0e0271608c87f618d86eb9c5f79046b277 Nov 02 22:10:31.038607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:31.038586 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:31.318323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:31.318282 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm"] Nov 02 22:10:31.425471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:31.425430 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" event={"ID":"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0","Type":"ContainerStarted","Data":"1528ae382498b5614955977ca0d5be8ae28f2fa89e22987c9db2c51c684a86bc"} Nov 02 22:10:31.427255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:31.427225 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" event={"ID":"135d60c1-4a7f-410d-962e-fe251e968897","Type":"ContainerStarted","Data":"9a1cf2fa345bb4510cd2be2cab687ddeccb24da5b68ccebf1331acc7d7fb1623"} Nov 02 22:10:31.427394 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:31.427287 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" event={"ID":"135d60c1-4a7f-410d-962e-fe251e968897","Type":"ContainerStarted","Data":"46d92404762ccc1bfdd53a5c4802ec0e0271608c87f618d86eb9c5f79046b277"} Nov 02 22:10:31.445690 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:31.445638 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" podStartSLOduration=1.445626342 podStartE2EDuration="1.445626342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:31.44133572 +0000 UTC m=+759.379448939" watchObservedRunningTime="2026-11-02 22:10:31.445626342 +0000 UTC m=+759.383739560" Nov 02 22:10:32.052825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.052790 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2"] Nov 02 22:10:32.056791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.056765 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.063771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.063432 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2"] Nov 02 22:10:32.148254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.148160 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwftx\" (UniqueName: \"kubernetes.io/projected/f01bc386-6c4e-42f3-9383-44b6ae4e6652-kube-api-access-fwftx\") pod \"maas-api-revoke-sub-164d9ca2-qfzx2\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.148254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.148218 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-164d9ca2-qfzx2\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.148465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.148305 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-164d9ca2-qfzx2\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.175455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.175419 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr"] Nov 02 22:10:32.179332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.179312 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.199710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.199286 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr"] Nov 02 22:10:32.249225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.249195 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwftx\" (UniqueName: \"kubernetes.io/projected/f01bc386-6c4e-42f3-9383-44b6ae4e6652-kube-api-access-fwftx\") pod \"maas-api-revoke-sub-164d9ca2-qfzx2\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.249390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.249242 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d89cfde-jbhvr\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.249390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.249279 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6m7\" (UniqueName: \"kubernetes.io/projected/0df1e741-13d0-4261-a1bb-4a0758d646cc-kube-api-access-2m6m7\") pod \"maas-api-revoke-sub-9d89cfde-jbhvr\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.249609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.249579 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-164d9ca2-qfzx2\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.249722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.249701 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-164d9ca2-qfzx2\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.249797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.249747 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d89cfde-jbhvr\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.250269 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.250246 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-164d9ca2-qfzx2\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.252106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.252082 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-164d9ca2-qfzx2\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.256813 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.256793 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwftx\" (UniqueName: \"kubernetes.io/projected/f01bc386-6c4e-42f3-9383-44b6ae4e6652-kube-api-access-fwftx\") pod \"maas-api-revoke-sub-164d9ca2-qfzx2\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.350526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.350482 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d89cfde-jbhvr\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.350706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.350536 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6m7\" (UniqueName: \"kubernetes.io/projected/0df1e741-13d0-4261-a1bb-4a0758d646cc-kube-api-access-2m6m7\") pod \"maas-api-revoke-sub-9d89cfde-jbhvr\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.350706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.350605 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d89cfde-jbhvr\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.351385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.351354 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d89cfde-jbhvr\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.352964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.352942 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d89cfde-jbhvr\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.359632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.359607 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6m7\" (UniqueName: \"kubernetes.io/projected/0df1e741-13d0-4261-a1bb-4a0758d646cc-kube-api-access-2m6m7\") pod \"maas-api-revoke-sub-9d89cfde-jbhvr\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.375031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.375009 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:32.446385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.446341 4309 generic.go:356] "Generic (PLEG): container finished" podID="9e6a24fc-3e68-4878-84bc-6051e9d2c1e0" containerID="c72152dfce2568d0a510f87cff750399b760499670c9c022f71b2d319d4cbc99" exitCode=0 Nov 02 22:10:32.446811 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.446432 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" event={"ID":"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0","Type":"ContainerDied","Data":"c72152dfce2568d0a510f87cff750399b760499670c9c022f71b2d319d4cbc99"} Nov 02 22:10:32.448996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.448966 4309 generic.go:356] "Generic (PLEG): container finished" podID="135d60c1-4a7f-410d-962e-fe251e968897" containerID="9a1cf2fa345bb4510cd2be2cab687ddeccb24da5b68ccebf1331acc7d7fb1623" exitCode=0 Nov 02 22:10:32.449121 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.449104 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" event={"ID":"135d60c1-4a7f-410d-962e-fe251e968897","Type":"ContainerDied","Data":"9a1cf2fa345bb4510cd2be2cab687ddeccb24da5b68ccebf1331acc7d7fb1623"} Nov 02 22:10:32.498696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.498642 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp"] Nov 02 22:10:32.504670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.504638 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.514332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.514304 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp"] Nov 02 22:10:32.554066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.554029 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587441d6-ktvkp\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.554306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.554103 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glwj9\" (UniqueName: \"kubernetes.io/projected/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-kube-api-access-glwj9\") pod \"maas-api-revoke-sub-587441d6-ktvkp\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.554306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.554181 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587441d6-ktvkp\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.560047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.560012 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:32.651533 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.651409 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2"] Nov 02 22:10:32.654516 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:32.654486 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01bc386_6c4e_42f3_9383_44b6ae4e6652.slice/crio-e6c06404b16dd4b97bcfe884037feb1621c795b3cdbe3948706303a13c0ca42b WatchSource:0}: Error finding container e6c06404b16dd4b97bcfe884037feb1621c795b3cdbe3948706303a13c0ca42b: Status 404 returned error can't find the container with id e6c06404b16dd4b97bcfe884037feb1621c795b3cdbe3948706303a13c0ca42b Nov 02 22:10:32.655116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.655090 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glwj9\" (UniqueName: \"kubernetes.io/projected/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-kube-api-access-glwj9\") pod \"maas-api-revoke-sub-587441d6-ktvkp\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.655210 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.655148 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587441d6-ktvkp\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.655356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.655340 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587441d6-ktvkp\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.656081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.656061 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-587441d6-ktvkp\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.659568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.659548 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-587441d6-ktvkp\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.662739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.662706 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glwj9\" (UniqueName: \"kubernetes.io/projected/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-kube-api-access-glwj9\") pod \"maas-api-revoke-sub-587441d6-ktvkp\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.827803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.827769 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:32.852121 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:32.852098 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr"] Nov 02 22:10:32.853542 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:32.853499 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df1e741_13d0_4261_a1bb_4a0758d646cc.slice/crio-77d1785fcd6221a74d66be527dcb75dd510619cd3fb0ea37321298de897e749e WatchSource:0}: Error finding container 77d1785fcd6221a74d66be527dcb75dd510619cd3fb0ea37321298de897e749e: Status 404 returned error can't find the container with id 77d1785fcd6221a74d66be527dcb75dd510619cd3fb0ea37321298de897e749e Nov 02 22:10:33.117215 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:33.117188 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d19c39_b1cf_49e8_ba55_2cf8b2d2f63d.slice/crio-11a925eccf121064a34c48fe89e155b42ba90add9b0e2a5479edc25396ff5a8b WatchSource:0}: Error finding container 11a925eccf121064a34c48fe89e155b42ba90add9b0e2a5479edc25396ff5a8b: Status 404 returned error can't find the container with id 11a925eccf121064a34c48fe89e155b42ba90add9b0e2a5479edc25396ff5a8b Nov 02 22:10:33.117529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.117497 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp"] Nov 02 22:10:33.457175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.457143 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" event={"ID":"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d","Type":"ContainerStarted","Data":"5febeb55bc2929182337f4ea0ef4951dfd14c3ef70c8bfb609f80377f11bfbf9"} Nov 02 22:10:33.457619 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.457202 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" event={"ID":"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d","Type":"ContainerStarted","Data":"11a925eccf121064a34c48fe89e155b42ba90add9b0e2a5479edc25396ff5a8b"} Nov 02 22:10:33.458839 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.458813 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" event={"ID":"0df1e741-13d0-4261-a1bb-4a0758d646cc","Type":"ContainerStarted","Data":"57f33735e211e89cdeee6eb116bb415bb2b6bac4a64532c3bbb727c10085d13c"} Nov 02 22:10:33.458946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.458859 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" event={"ID":"0df1e741-13d0-4261-a1bb-4a0758d646cc","Type":"ContainerStarted","Data":"77d1785fcd6221a74d66be527dcb75dd510619cd3fb0ea37321298de897e749e"} Nov 02 22:10:33.460628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.460589 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" event={"ID":"f01bc386-6c4e-42f3-9383-44b6ae4e6652","Type":"ContainerStarted","Data":"ea1e5839def8ad37c0b6045dbdbff175949396a21e3d6b069c2741c9d0763be7"} Nov 02 22:10:33.460733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.460642 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" event={"ID":"f01bc386-6c4e-42f3-9383-44b6ae4e6652","Type":"ContainerStarted","Data":"e6c06404b16dd4b97bcfe884037feb1621c795b3cdbe3948706303a13c0ca42b"} Nov 02 22:10:33.476179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.476130 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" podStartSLOduration=1.476114088 podStartE2EDuration="1.476114088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:33.473021129 +0000 UTC m=+761.411134370" watchObservedRunningTime="2026-11-02 22:10:33.476114088 +0000 UTC m=+761.414227305" Nov 02 22:10:33.490667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.490612 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" podStartSLOduration=1.490598971 podStartE2EDuration="1.490598971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:33.488976973 +0000 UTC m=+761.427090193" watchObservedRunningTime="2026-11-02 22:10:33.490598971 +0000 UTC m=+761.428712190" Nov 02 22:10:33.515138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.514787 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" podStartSLOduration=1.514766303 podStartE2EDuration="1.514766303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:33.509376016 +0000 UTC m=+761.447489237" watchObservedRunningTime="2026-11-02 22:10:33.514766303 +0000 UTC m=+761.452879522" Nov 02 22:10:33.681461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.681440 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:33.698883 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.698853 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:33.769075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.769037 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-openshift-ca\") pod \"135d60c1-4a7f-410d-962e-fe251e968897\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " Nov 02 22:10:33.769248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.769102 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-kube-api-access-c2ssp\" (UniqueName: \"kubernetes.io/projected/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-kube-api-access-c2ssp\") pod \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " Nov 02 22:10:33.769248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.769194 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-xks-ca\") pod \"135d60c1-4a7f-410d-962e-fe251e968897\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " Nov 02 22:10:33.769248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.769227 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135d60c1-4a7f-410d-962e-fe251e968897-kube-api-access-srltm\" (UniqueName: \"kubernetes.io/projected/135d60c1-4a7f-410d-962e-fe251e968897-kube-api-access-srltm\") pod \"135d60c1-4a7f-410d-962e-fe251e968897\" (UID: \"135d60c1-4a7f-410d-962e-fe251e968897\") " Nov 02 22:10:33.769404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.769293 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-openshift-ca\") pod \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " Nov 02 22:10:33.769404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.769358 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-xks-ca\") pod \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\" (UID: \"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0\") " Nov 02 22:10:33.769534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.769445 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-openshift-ca" pod "135d60c1-4a7f-410d-962e-fe251e968897" (UID: "135d60c1-4a7f-410d-962e-fe251e968897"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:33.769860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.769836 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:33.770330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.770300 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-openshift-ca" pod "9e6a24fc-3e68-4878-84bc-6051e9d2c1e0" (UID: "9e6a24fc-3e68-4878-84bc-6051e9d2c1e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:33.771835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.771803 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135d60c1-4a7f-410d-962e-fe251e968897-kube-api-access-srltm" pod "135d60c1-4a7f-410d-962e-fe251e968897" (UID: "135d60c1-4a7f-410d-962e-fe251e968897"). InnerVolumeSpecName "kube-api-access-srltm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:33.772104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.772081 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-xks-ca" pod "9e6a24fc-3e68-4878-84bc-6051e9d2c1e0" (UID: "9e6a24fc-3e68-4878-84bc-6051e9d2c1e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:33.772193 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.772170 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-kube-api-access-c2ssp" pod "9e6a24fc-3e68-4878-84bc-6051e9d2c1e0" (UID: "9e6a24fc-3e68-4878-84bc-6051e9d2c1e0"). InnerVolumeSpecName "kube-api-access-c2ssp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:33.772193 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.772156 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-xks-ca" pod "135d60c1-4a7f-410d-962e-fe251e968897" (UID: "135d60c1-4a7f-410d-962e-fe251e968897"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:33.836515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.836428 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z"] Nov 02 22:10:33.837076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.837059 4309 cpu_manager.go:414] "containerMap: removing container" podUID="9e6a24fc-3e68-4878-84bc-6051e9d2c1e0" containerName="revoke-keys" Nov 02 22:10:33.837161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.837077 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6a24fc-3e68-4878-84bc-6051e9d2c1e0" containerName="revoke-keys" Nov 02 22:10:33.837161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.837102 4309 cpu_manager.go:414] "containerMap: removing container" podUID="135d60c1-4a7f-410d-962e-fe251e968897" containerName="revoke-keys" Nov 02 22:10:33.837161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.837107 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135d60c1-4a7f-410d-962e-fe251e968897" containerName="revoke-keys" Nov 02 22:10:33.837308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.837207 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="135d60c1-4a7f-410d-962e-fe251e968897" containerName="revoke-keys" Nov 02 22:10:33.837308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.837222 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6a24fc-3e68-4878-84bc-6051e9d2c1e0" containerName="revoke-keys" Nov 02 22:10:33.840948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.840925 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:33.842894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.842862 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z"] Nov 02 22:10:33.871320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.871292 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2ssp\" (UniqueName: \"kubernetes.io/projected/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-kube-api-access-c2ssp\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:33.871320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.871322 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d60c1-4a7f-410d-962e-fe251e968897-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:33.871477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.871332 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srltm\" (UniqueName: \"kubernetes.io/projected/135d60c1-4a7f-410d-962e-fe251e968897-kube-api-access-srltm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:33.871477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.871340 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:33.871477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.871353 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:33.972659 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.972622 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b1fd6e-qhf7z\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:33.972817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.972701 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d5f9\" (UniqueName: \"kubernetes.io/projected/267967a3-2066-4b27-9afa-9b8cdab1788a-kube-api-access-8d5f9\") pod \"maas-api-revoke-sub-b6b1fd6e-qhf7z\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:33.972817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:33.972770 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b1fd6e-qhf7z\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:34.073508 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.073473 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b1fd6e-qhf7z\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:34.073719 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.073533 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d5f9\" (UniqueName: \"kubernetes.io/projected/267967a3-2066-4b27-9afa-9b8cdab1788a-kube-api-access-8d5f9\") pod \"maas-api-revoke-sub-b6b1fd6e-qhf7z\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:34.073719 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.073557 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b1fd6e-qhf7z\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:34.074342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.074319 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b1fd6e-qhf7z\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:34.075948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.075925 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b1fd6e-qhf7z\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:34.081093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.081067 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d5f9\" (UniqueName: \"kubernetes.io/projected/267967a3-2066-4b27-9afa-9b8cdab1788a-kube-api-access-8d5f9\") pod \"maas-api-revoke-sub-b6b1fd6e-qhf7z\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:34.167915 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.167831 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:34.454696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.454652 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z"] Nov 02 22:10:34.455556 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:34.455522 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267967a3_2066_4b27_9afa_9b8cdab1788a.slice/crio-a50fb0d1590df1a9c38f4f3ca2432dd54316fcb3c08bb9a38a0d932141fde72a WatchSource:0}: Error finding container a50fb0d1590df1a9c38f4f3ca2432dd54316fcb3c08bb9a38a0d932141fde72a: Status 404 returned error can't find the container with id a50fb0d1590df1a9c38f4f3ca2432dd54316fcb3c08bb9a38a0d932141fde72a Nov 02 22:10:34.467438 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.467415 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" Nov 02 22:10:34.467802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.467414 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm" event={"ID":"9e6a24fc-3e68-4878-84bc-6051e9d2c1e0","Type":"ContainerDied","Data":"1528ae382498b5614955977ca0d5be8ae28f2fa89e22987c9db2c51c684a86bc"} Nov 02 22:10:34.467802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.467547 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1528ae382498b5614955977ca0d5be8ae28f2fa89e22987c9db2c51c684a86bc" Nov 02 22:10:34.468994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.468974 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" Nov 02 22:10:34.469100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.468994 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h" event={"ID":"135d60c1-4a7f-410d-962e-fe251e968897","Type":"ContainerDied","Data":"46d92404762ccc1bfdd53a5c4802ec0e0271608c87f618d86eb9c5f79046b277"} Nov 02 22:10:34.469100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.469038 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46d92404762ccc1bfdd53a5c4802ec0e0271608c87f618d86eb9c5f79046b277" Nov 02 22:10:34.471170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.471010 4309 generic.go:356] "Generic (PLEG): container finished" podID="f01bc386-6c4e-42f3-9383-44b6ae4e6652" containerID="ea1e5839def8ad37c0b6045dbdbff175949396a21e3d6b069c2741c9d0763be7" exitCode=0 Nov 02 22:10:34.471170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.471084 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" event={"ID":"f01bc386-6c4e-42f3-9383-44b6ae4e6652","Type":"ContainerDied","Data":"ea1e5839def8ad37c0b6045dbdbff175949396a21e3d6b069c2741c9d0763be7"} Nov 02 22:10:34.473090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.473072 4309 generic.go:356] "Generic (PLEG): container finished" podID="a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d" containerID="5febeb55bc2929182337f4ea0ef4951dfd14c3ef70c8bfb609f80377f11bfbf9" exitCode=0 Nov 02 22:10:34.473191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.473144 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" event={"ID":"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d","Type":"ContainerDied","Data":"5febeb55bc2929182337f4ea0ef4951dfd14c3ef70c8bfb609f80377f11bfbf9"} Nov 02 22:10:34.474954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.474926 4309 generic.go:356] "Generic (PLEG): container finished" podID="0df1e741-13d0-4261-a1bb-4a0758d646cc" containerID="57f33735e211e89cdeee6eb116bb415bb2b6bac4a64532c3bbb727c10085d13c" exitCode=0 Nov 02 22:10:34.475072 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.475029 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" event={"ID":"0df1e741-13d0-4261-a1bb-4a0758d646cc","Type":"ContainerDied","Data":"57f33735e211e89cdeee6eb116bb415bb2b6bac4a64532c3bbb727c10085d13c"} Nov 02 22:10:34.476568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.476548 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" event={"ID":"267967a3-2066-4b27-9afa-9b8cdab1788a","Type":"ContainerStarted","Data":"a50fb0d1590df1a9c38f4f3ca2432dd54316fcb3c08bb9a38a0d932141fde72a"} Nov 02 22:10:34.959357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.959327 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s"] Nov 02 22:10:34.963342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.963322 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:34.970554 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:34.970528 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s"] Nov 02 22:10:35.085098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.085025 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e7ca20e-8jb2s\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.085098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.085068 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d272b\" (UniqueName: \"kubernetes.io/projected/40e8a5ef-1a26-4024-abca-6abe843850bf-kube-api-access-d272b\") pod \"maas-api-revoke-sub-0e7ca20e-8jb2s\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.085268 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.085218 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e7ca20e-8jb2s\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.186356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.186274 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e7ca20e-8jb2s\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.186498 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.186388 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e7ca20e-8jb2s\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.186498 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.186418 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d272b\" (UniqueName: \"kubernetes.io/projected/40e8a5ef-1a26-4024-abca-6abe843850bf-kube-api-access-d272b\") pod \"maas-api-revoke-sub-0e7ca20e-8jb2s\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.187092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.187064 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e7ca20e-8jb2s\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.188598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.188580 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e7ca20e-8jb2s\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.194089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.194066 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d272b\" (UniqueName: \"kubernetes.io/projected/40e8a5ef-1a26-4024-abca-6abe843850bf-kube-api-access-d272b\") pod \"maas-api-revoke-sub-0e7ca20e-8jb2s\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.279950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.279919 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:35.493629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.493527 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" event={"ID":"267967a3-2066-4b27-9afa-9b8cdab1788a","Type":"ContainerStarted","Data":"7329f6ee84d03a813bf2f78c4f2f9baffe6a810e2e94cbef95ffaa192f5dc59b"} Nov 02 22:10:35.512717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.511666 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" podStartSLOduration=2.511647217 podStartE2EDuration="2.511647217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:35.509172334 +0000 UTC m=+763.447285553" watchObservedRunningTime="2026-11-02 22:10:35.511647217 +0000 UTC m=+763.449760435" Nov 02 22:10:35.560885 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.560854 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s"] Nov 02 22:10:35.742850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.742827 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:35.745509 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.745452 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:35.753911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.753890 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:35.894172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894139 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-openshift-ca\") pod \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " Nov 02 22:10:35.894359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894246 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-xks-ca\") pod \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " Nov 02 22:10:35.894359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894308 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df1e741-13d0-4261-a1bb-4a0758d646cc-kube-api-access-2m6m7\" (UniqueName: \"kubernetes.io/projected/0df1e741-13d0-4261-a1bb-4a0758d646cc-kube-api-access-2m6m7\") pod \"0df1e741-13d0-4261-a1bb-4a0758d646cc\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " Nov 02 22:10:35.894359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894353 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-openshift-ca\") pod \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " Nov 02 22:10:35.894539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894415 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01bc386-6c4e-42f3-9383-44b6ae4e6652-kube-api-access-fwftx\" (UniqueName: \"kubernetes.io/projected/f01bc386-6c4e-42f3-9383-44b6ae4e6652-kube-api-access-fwftx\") pod \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\" (UID: \"f01bc386-6c4e-42f3-9383-44b6ae4e6652\") " Nov 02 22:10:35.894539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894443 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-xks-ca\") pod \"0df1e741-13d0-4261-a1bb-4a0758d646cc\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " Nov 02 22:10:35.894539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894488 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-kube-api-access-glwj9\" (UniqueName: \"kubernetes.io/projected/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-kube-api-access-glwj9\") pod \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " Nov 02 22:10:35.894539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894528 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-openshift-ca\") pod \"0df1e741-13d0-4261-a1bb-4a0758d646cc\" (UID: \"0df1e741-13d0-4261-a1bb-4a0758d646cc\") " Nov 02 22:10:35.894783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894540 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-openshift-ca" pod "a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d" (UID: "a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:35.894783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.894639 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-xks-ca\") pod \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\" (UID: \"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d\") " Nov 02 22:10:35.895195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.895156 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:35.895560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.895529 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-openshift-ca" pod "f01bc386-6c4e-42f3-9383-44b6ae4e6652" (UID: "f01bc386-6c4e-42f3-9383-44b6ae4e6652"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:35.895704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.895663 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-openshift-ca" pod "0df1e741-13d0-4261-a1bb-4a0758d646cc" (UID: "0df1e741-13d0-4261-a1bb-4a0758d646cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:35.897328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.897304 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-xks-ca" pod "0df1e741-13d0-4261-a1bb-4a0758d646cc" (UID: "0df1e741-13d0-4261-a1bb-4a0758d646cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:35.897574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.897517 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-xks-ca" pod "a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d" (UID: "a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:35.897997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.897968 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df1e741-13d0-4261-a1bb-4a0758d646cc-kube-api-access-2m6m7" pod "0df1e741-13d0-4261-a1bb-4a0758d646cc" (UID: "0df1e741-13d0-4261-a1bb-4a0758d646cc"). InnerVolumeSpecName "kube-api-access-2m6m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:35.897997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.897985 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01bc386-6c4e-42f3-9383-44b6ae4e6652-kube-api-access-fwftx" pod "f01bc386-6c4e-42f3-9383-44b6ae4e6652" (UID: "f01bc386-6c4e-42f3-9383-44b6ae4e6652"). InnerVolumeSpecName "kube-api-access-fwftx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:35.898893 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.898866 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-kube-api-access-glwj9" pod "a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d" (UID: "a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d"). InnerVolumeSpecName "kube-api-access-glwj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:35.899266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.899247 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-xks-ca" pod "f01bc386-6c4e-42f3-9383-44b6ae4e6652" (UID: "f01bc386-6c4e-42f3-9383-44b6ae4e6652"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:35.996665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.996600 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:35.996665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.996626 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:35.996665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.996636 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:35.996665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.996645 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m6m7\" (UniqueName: \"kubernetes.io/projected/0df1e741-13d0-4261-a1bb-4a0758d646cc-kube-api-access-2m6m7\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:35.996665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.996653 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f01bc386-6c4e-42f3-9383-44b6ae4e6652-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:35.996665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.996663 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwftx\" (UniqueName: \"kubernetes.io/projected/f01bc386-6c4e-42f3-9383-44b6ae4e6652-kube-api-access-fwftx\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:35.996970 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.996686 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0df1e741-13d0-4261-a1bb-4a0758d646cc-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:35.996970 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:35.996695 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glwj9\" (UniqueName: \"kubernetes.io/projected/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d-kube-api-access-glwj9\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:36.504699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.504524 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" event={"ID":"a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d","Type":"ContainerDied","Data":"11a925eccf121064a34c48fe89e155b42ba90add9b0e2a5479edc25396ff5a8b"} Nov 02 22:10:36.504699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.504591 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11a925eccf121064a34c48fe89e155b42ba90add9b0e2a5479edc25396ff5a8b" Nov 02 22:10:36.505488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.505438 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp" Nov 02 22:10:36.508530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.508501 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" event={"ID":"40e8a5ef-1a26-4024-abca-6abe843850bf","Type":"ContainerStarted","Data":"0934ab8ad24bac6983770609626f59f619eee6dbb72faaf5c77ea2fc1d7240c5"} Nov 02 22:10:36.508645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.508557 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" event={"ID":"40e8a5ef-1a26-4024-abca-6abe843850bf","Type":"ContainerStarted","Data":"609de8c59cef9754cfe593065af43748051efc5438672ad334e7ab6ac848fdf7"} Nov 02 22:10:36.511263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.511232 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" event={"ID":"0df1e741-13d0-4261-a1bb-4a0758d646cc","Type":"ContainerDied","Data":"77d1785fcd6221a74d66be527dcb75dd510619cd3fb0ea37321298de897e749e"} Nov 02 22:10:36.511384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.511297 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77d1785fcd6221a74d66be527dcb75dd510619cd3fb0ea37321298de897e749e" Nov 02 22:10:36.511822 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.511801 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr" Nov 02 22:10:36.514803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.514768 4309 generic.go:356] "Generic (PLEG): container finished" podID="267967a3-2066-4b27-9afa-9b8cdab1788a" containerID="7329f6ee84d03a813bf2f78c4f2f9baffe6a810e2e94cbef95ffaa192f5dc59b" exitCode=0 Nov 02 22:10:36.515054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.515028 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" event={"ID":"267967a3-2066-4b27-9afa-9b8cdab1788a","Type":"ContainerDied","Data":"7329f6ee84d03a813bf2f78c4f2f9baffe6a810e2e94cbef95ffaa192f5dc59b"} Nov 02 22:10:36.517400 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.517377 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" event={"ID":"f01bc386-6c4e-42f3-9383-44b6ae4e6652","Type":"ContainerDied","Data":"e6c06404b16dd4b97bcfe884037feb1621c795b3cdbe3948706303a13c0ca42b"} Nov 02 22:10:36.517495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.517422 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6c06404b16dd4b97bcfe884037feb1621c795b3cdbe3948706303a13c0ca42b" Nov 02 22:10:36.517495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.517472 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2" Nov 02 22:10:36.529483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:36.529443 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" podStartSLOduration=2.529431867 podStartE2EDuration="2.529431867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:36.525053003 +0000 UTC m=+764.463166221" watchObservedRunningTime="2026-11-02 22:10:36.529431867 +0000 UTC m=+764.467545084" Nov 02 22:10:37.541532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:37.541486 4309 generic.go:356] "Generic (PLEG): container finished" podID="40e8a5ef-1a26-4024-abca-6abe843850bf" containerID="0934ab8ad24bac6983770609626f59f619eee6dbb72faaf5c77ea2fc1d7240c5" exitCode=0 Nov 02 22:10:37.542780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:37.542752 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" event={"ID":"40e8a5ef-1a26-4024-abca-6abe843850bf","Type":"ContainerDied","Data":"0934ab8ad24bac6983770609626f59f619eee6dbb72faaf5c77ea2fc1d7240c5"} Nov 02 22:10:38.108697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.108122 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:38.174123 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.174085 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-openshift-ca\") pod \"267967a3-2066-4b27-9afa-9b8cdab1788a\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " Nov 02 22:10:38.174310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.174206 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-xks-ca\") pod \"267967a3-2066-4b27-9afa-9b8cdab1788a\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " Nov 02 22:10:38.174310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.174296 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267967a3-2066-4b27-9afa-9b8cdab1788a-kube-api-access-8d5f9\" (UniqueName: \"kubernetes.io/projected/267967a3-2066-4b27-9afa-9b8cdab1788a-kube-api-access-8d5f9\") pod \"267967a3-2066-4b27-9afa-9b8cdab1788a\" (UID: \"267967a3-2066-4b27-9afa-9b8cdab1788a\") " Nov 02 22:10:38.175649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.175615 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-openshift-ca" pod "267967a3-2066-4b27-9afa-9b8cdab1788a" (UID: "267967a3-2066-4b27-9afa-9b8cdab1788a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:38.178617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.177392 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267967a3-2066-4b27-9afa-9b8cdab1788a-kube-api-access-8d5f9" pod "267967a3-2066-4b27-9afa-9b8cdab1788a" (UID: "267967a3-2066-4b27-9afa-9b8cdab1788a"). InnerVolumeSpecName "kube-api-access-8d5f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:38.187043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.186971 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-xks-ca" pod "267967a3-2066-4b27-9afa-9b8cdab1788a" (UID: "267967a3-2066-4b27-9afa-9b8cdab1788a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:38.275701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.275640 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:38.275701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.275702 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d5f9\" (UniqueName: \"kubernetes.io/projected/267967a3-2066-4b27-9afa-9b8cdab1788a-kube-api-access-8d5f9\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:38.275934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.275719 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/267967a3-2066-4b27-9afa-9b8cdab1788a-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:38.554984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.554942 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" event={"ID":"267967a3-2066-4b27-9afa-9b8cdab1788a","Type":"ContainerDied","Data":"a50fb0d1590df1a9c38f4f3ca2432dd54316fcb3c08bb9a38a0d932141fde72a"} Nov 02 22:10:38.555468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.555001 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a50fb0d1590df1a9c38f4f3ca2432dd54316fcb3c08bb9a38a0d932141fde72a" Nov 02 22:10:38.555468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.555113 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z" Nov 02 22:10:38.859150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.859080 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:38.945007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.944976 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-xks-ca\") pod \"40e8a5ef-1a26-4024-abca-6abe843850bf\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " Nov 02 22:10:38.945249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.945230 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e8a5ef-1a26-4024-abca-6abe843850bf-kube-api-access-d272b\" (UniqueName: \"kubernetes.io/projected/40e8a5ef-1a26-4024-abca-6abe843850bf-kube-api-access-d272b\") pod \"40e8a5ef-1a26-4024-abca-6abe843850bf\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " Nov 02 22:10:38.945442 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.945425 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-openshift-ca\") pod \"40e8a5ef-1a26-4024-abca-6abe843850bf\" (UID: \"40e8a5ef-1a26-4024-abca-6abe843850bf\") " Nov 02 22:10:38.946861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.946837 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-openshift-ca" pod "40e8a5ef-1a26-4024-abca-6abe843850bf" (UID: "40e8a5ef-1a26-4024-abca-6abe843850bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:38.949805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.949768 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-xks-ca" pod "40e8a5ef-1a26-4024-abca-6abe843850bf" (UID: "40e8a5ef-1a26-4024-abca-6abe843850bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:38.952340 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:38.952306 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e8a5ef-1a26-4024-abca-6abe843850bf-kube-api-access-d272b" pod "40e8a5ef-1a26-4024-abca-6abe843850bf" (UID: "40e8a5ef-1a26-4024-abca-6abe843850bf"). InnerVolumeSpecName "kube-api-access-d272b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:39.048819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:39.048741 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:39.048819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:39.048777 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d272b\" (UniqueName: \"kubernetes.io/projected/40e8a5ef-1a26-4024-abca-6abe843850bf-kube-api-access-d272b\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:39.048819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:39.048794 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e8a5ef-1a26-4024-abca-6abe843850bf-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:39.567284 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:39.567256 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" Nov 02 22:10:39.567738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:39.567286 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s" event={"ID":"40e8a5ef-1a26-4024-abca-6abe843850bf","Type":"ContainerDied","Data":"609de8c59cef9754cfe593065af43748051efc5438672ad334e7ab6ac848fdf7"} Nov 02 22:10:39.567738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:39.567337 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="609de8c59cef9754cfe593065af43748051efc5438672ad334e7ab6ac848fdf7" Nov 02 22:10:41.402586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.402544 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89"] Nov 02 22:10:41.403514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403487 4309 cpu_manager.go:414] "containerMap: removing container" podUID="40e8a5ef-1a26-4024-abca-6abe843850bf" containerName="revoke-keys" Nov 02 22:10:41.403611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403515 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e8a5ef-1a26-4024-abca-6abe843850bf" containerName="revoke-keys" Nov 02 22:10:41.403611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403534 4309 cpu_manager.go:414] "containerMap: removing container" podUID="a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d" containerName="revoke-keys" Nov 02 22:10:41.403611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403543 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d" containerName="revoke-keys" Nov 02 22:10:41.403611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403555 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f01bc386-6c4e-42f3-9383-44b6ae4e6652" containerName="revoke-keys" Nov 02 22:10:41.403611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403562 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01bc386-6c4e-42f3-9383-44b6ae4e6652" containerName="revoke-keys" Nov 02 22:10:41.403611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403589 4309 cpu_manager.go:414] "containerMap: removing container" podUID="0df1e741-13d0-4261-a1bb-4a0758d646cc" containerName="revoke-keys" Nov 02 22:10:41.403611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403596 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df1e741-13d0-4261-a1bb-4a0758d646cc" containerName="revoke-keys" Nov 02 22:10:41.403611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403611 4309 cpu_manager.go:414] "containerMap: removing container" podUID="267967a3-2066-4b27-9afa-9b8cdab1788a" containerName="revoke-keys" Nov 02 22:10:41.403895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403618 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267967a3-2066-4b27-9afa-9b8cdab1788a" containerName="revoke-keys" Nov 02 22:10:41.403935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403889 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e8a5ef-1a26-4024-abca-6abe843850bf" containerName="revoke-keys" Nov 02 22:10:41.403935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403908 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d" containerName="revoke-keys" Nov 02 22:10:41.403935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403920 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01bc386-6c4e-42f3-9383-44b6ae4e6652" containerName="revoke-keys" Nov 02 22:10:41.403935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403930 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df1e741-13d0-4261-a1bb-4a0758d646cc" containerName="revoke-keys" Nov 02 22:10:41.404127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.403943 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="267967a3-2066-4b27-9afa-9b8cdab1788a" containerName="revoke-keys" Nov 02 22:10:41.412001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.411977 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.413297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.413274 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89"] Nov 02 22:10:41.420809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.420785 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2532af\"/\"default-dockercfg-wgtlm\"" Nov 02 22:10:41.421026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.420811 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2532af\"/\"openshift-service-ca.crt\"" Nov 02 22:10:41.422729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.420933 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2532af\"/\"distinct-w2-2532af-kserve-self-signed-certs\"" Nov 02 22:10:41.422890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.421429 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2532af\"/\"kube-root-ca.crt\"" Nov 02 22:10:41.580053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.580012 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7gx\" (UniqueName: \"kubernetes.io/projected/d00f0e87-7086-4d10-b9a4-713cc4948990-kube-api-access-jj7gx\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.580280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.580187 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d00f0e87-7086-4d10-b9a4-713cc4948990-tls-certs\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.580280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.580237 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-home\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.580401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.580283 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-model-cache\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.580401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.580314 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-dshm\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.580401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.580345 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-tmp-dir\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.602982 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:10:41.602933 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:10:41.681814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.681717 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7gx\" (UniqueName: \"kubernetes.io/projected/d00f0e87-7086-4d10-b9a4-713cc4948990-kube-api-access-jj7gx\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.681984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.681819 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d00f0e87-7086-4d10-b9a4-713cc4948990-tls-certs\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.681984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.681844 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-home\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.681984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.681872 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-model-cache\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.681984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.681892 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-dshm\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.681984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.681921 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-tmp-dir\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.682354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.682323 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-home\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.682505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.682398 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-tmp-dir\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.682567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.682547 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-model-cache\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.684178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.684158 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-dshm\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.684893 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.684871 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d00f0e87-7086-4d10-b9a4-713cc4948990-tls-certs\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.689294 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.689267 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7gx\" (UniqueName: \"kubernetes.io/projected/d00f0e87-7086-4d10-b9a4-713cc4948990-kube-api-access-jj7gx\") pod \"distinct-w2-2532af-kserve-746c6865b4-xdw89\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:41.729183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:41.729150 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:42.012691 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.012650 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89"] Nov 02 22:10:42.013345 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:42.013313 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd00f0e87_7086_4d10_b9a4_713cc4948990.slice/crio-21809fb6096eb7f558c44acf15d905dfb85efe248ae77d65d01f5dd8222928f7 WatchSource:0}: Error finding container 21809fb6096eb7f558c44acf15d905dfb85efe248ae77d65d01f5dd8222928f7: Status 404 returned error can't find the container with id 21809fb6096eb7f558c44acf15d905dfb85efe248ae77d65d01f5dd8222928f7 Nov 02 22:10:42.591468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.591430 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" event={"ID":"d00f0e87-7086-4d10-b9a4-713cc4948990","Type":"ContainerStarted","Data":"70fd2577c2985484420e6e4bbd94d17d19140befbc0b28810372bf178857c855"} Nov 02 22:10:42.591468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.591477 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" event={"ID":"d00f0e87-7086-4d10-b9a4-713cc4948990","Type":"ContainerStarted","Data":"21809fb6096eb7f558c44acf15d905dfb85efe248ae77d65d01f5dd8222928f7"} Nov 02 22:10:42.610404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.610348 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podStartSLOduration=1.610329111 podStartE2EDuration="1.610329111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:42.606778823 +0000 UTC m=+770.544892042" watchObservedRunningTime="2026-11-02 22:10:42.610329111 +0000 UTC m=+770.548442330" Nov 02 22:10:42.730223 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.730194 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:42.822404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.822368 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f"] Nov 02 22:10:42.826588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.826567 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:42.832977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.832954 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:10:42.833166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.833018 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:10:42.833166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.833009 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:10:42.833343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.833064 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:10:42.838534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:42.838503 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f"] Nov 02 22:10:43.002488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.002451 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrr2\" (UniqueName: \"kubernetes.io/projected/62f55ca6-8893-47ce-a6ed-c1357d1159ef-kube-api-access-mjrr2\") pod \"maas-api-revoke-sub-73a2d826-nhl6f\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.002751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.002716 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73a2d826-nhl6f\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.002818 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.002764 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73a2d826-nhl6f\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.104146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.104043 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73a2d826-nhl6f\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.104146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.104087 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73a2d826-nhl6f\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.104362 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.104212 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrr2\" (UniqueName: \"kubernetes.io/projected/62f55ca6-8893-47ce-a6ed-c1357d1159ef-kube-api-access-mjrr2\") pod \"maas-api-revoke-sub-73a2d826-nhl6f\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.105158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.105126 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73a2d826-nhl6f\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.106575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.106550 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73a2d826-nhl6f\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.112889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.112865 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrr2\" (UniqueName: \"kubernetes.io/projected/62f55ca6-8893-47ce-a6ed-c1357d1159ef-kube-api-access-mjrr2\") pod \"maas-api-revoke-sub-73a2d826-nhl6f\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.152007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.151971 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:43.434323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.434295 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f"] Nov 02 22:10:43.435895 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:43.435866 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f55ca6_8893_47ce_a6ed_c1357d1159ef.slice/crio-42dfe6340756bd2db2c0ca55552d120e39159b5fab0120d2b1119edec5ef0ae8 WatchSource:0}: Error finding container 42dfe6340756bd2db2c0ca55552d120e39159b5fab0120d2b1119edec5ef0ae8: Status 404 returned error can't find the container with id 42dfe6340756bd2db2c0ca55552d120e39159b5fab0120d2b1119edec5ef0ae8 Nov 02 22:10:43.602891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:43.602855 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" event={"ID":"62f55ca6-8893-47ce-a6ed-c1357d1159ef","Type":"ContainerStarted","Data":"42dfe6340756bd2db2c0ca55552d120e39159b5fab0120d2b1119edec5ef0ae8"} Nov 02 22:10:44.612042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:44.612006 4309 generic.go:356] "Generic (PLEG): container finished" podID="62f55ca6-8893-47ce-a6ed-c1357d1159ef" containerID="b805d49709dbbffd5acdbad93af9e027aaf48760c8d86762b49f526034fb9363" exitCode=0 Nov 02 22:10:44.612462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:44.612103 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" event={"ID":"62f55ca6-8893-47ce-a6ed-c1357d1159ef","Type":"ContainerDied","Data":"b805d49709dbbffd5acdbad93af9e027aaf48760c8d86762b49f526034fb9363"} Nov 02 22:10:45.128995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.128957 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l"] Nov 02 22:10:45.133004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.132981 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.140130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.140106 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l"] Nov 02 22:10:45.170925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.170896 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b"] Nov 02 22:10:45.176042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.175452 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.177543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.177505 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b"] Nov 02 22:10:45.187557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.187531 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62"] Nov 02 22:10:45.192325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.192304 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.196860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.196828 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62"] Nov 02 22:10:45.228559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.228525 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-904c2513-x2m7l\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.228746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.228691 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmdsk\" (UniqueName: \"kubernetes.io/projected/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-kube-api-access-jmdsk\") pod \"maas-api-revoke-sub-904c2513-x2m7l\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.228821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.228750 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-904c2513-x2m7l\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.329745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.329703 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5426dfd9-pbw9b\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.329921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.329851 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea7b4c4-qwk62\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.329921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.329883 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5426dfd9-pbw9b\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.329921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.329915 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmdsk\" (UniqueName: \"kubernetes.io/projected/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-kube-api-access-jmdsk\") pod \"maas-api-revoke-sub-904c2513-x2m7l\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.330080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.330046 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-904c2513-x2m7l\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.330144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.330096 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txqzb\" (UniqueName: \"kubernetes.io/projected/71ea6617-8b17-44c5-825c-2f5e4d80f388-kube-api-access-txqzb\") pod \"maas-api-revoke-sub-fea7b4c4-qwk62\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.330181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.330154 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j299\" (UniqueName: \"kubernetes.io/projected/779db660-69dc-4cf4-a004-5b68ba24cdea-kube-api-access-6j299\") pod \"maas-api-revoke-sub-5426dfd9-pbw9b\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.330214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.330190 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea7b4c4-qwk62\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.330335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.330315 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-904c2513-x2m7l\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.330967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.330938 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-904c2513-x2m7l\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.332475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.332452 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-904c2513-x2m7l\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.338251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.338225 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmdsk\" (UniqueName: \"kubernetes.io/projected/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-kube-api-access-jmdsk\") pod \"maas-api-revoke-sub-904c2513-x2m7l\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.431878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.431842 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5426dfd9-pbw9b\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.432057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.431958 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea7b4c4-qwk62\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.432057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.431991 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5426dfd9-pbw9b\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.432186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.432088 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txqzb\" (UniqueName: \"kubernetes.io/projected/71ea6617-8b17-44c5-825c-2f5e4d80f388-kube-api-access-txqzb\") pod \"maas-api-revoke-sub-fea7b4c4-qwk62\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.432186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.432115 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j299\" (UniqueName: \"kubernetes.io/projected/779db660-69dc-4cf4-a004-5b68ba24cdea-kube-api-access-6j299\") pod \"maas-api-revoke-sub-5426dfd9-pbw9b\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.432186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.432150 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea7b4c4-qwk62\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.432829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.432794 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5426dfd9-pbw9b\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.433020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.432999 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fea7b4c4-qwk62\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.434480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.434447 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fea7b4c4-qwk62\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.434635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.434589 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5426dfd9-pbw9b\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.439482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.439462 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j299\" (UniqueName: \"kubernetes.io/projected/779db660-69dc-4cf4-a004-5b68ba24cdea-kube-api-access-6j299\") pod \"maas-api-revoke-sub-5426dfd9-pbw9b\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.439595 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.439493 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txqzb\" (UniqueName: \"kubernetes.io/projected/71ea6617-8b17-44c5-825c-2f5e4d80f388-kube-api-access-txqzb\") pod \"maas-api-revoke-sub-fea7b4c4-qwk62\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.451170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.451144 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:45.491647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.491618 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:45.542516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.541238 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:45.745594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.745561 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l"] Nov 02 22:10:45.750016 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:45.749988 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ceaf50_e4ab_4ccb_b5b5_fad5e395669d.slice/crio-1e29895992d1e5d57f82b56746d205dd9fb081a89b0f3c96ad7f46111a4dc6bc WatchSource:0}: Error finding container 1e29895992d1e5d57f82b56746d205dd9fb081a89b0f3c96ad7f46111a4dc6bc: Status 404 returned error can't find the container with id 1e29895992d1e5d57f82b56746d205dd9fb081a89b0f3c96ad7f46111a4dc6bc Nov 02 22:10:45.789098 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:45.789071 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779db660_69dc_4cf4_a004_5b68ba24cdea.slice/crio-12bf90b49de01c2ee09394cc06cd45d6e228827305ea5858ffc756984225dad4 WatchSource:0}: Error finding container 12bf90b49de01c2ee09394cc06cd45d6e228827305ea5858ffc756984225dad4: Status 404 returned error can't find the container with id 12bf90b49de01c2ee09394cc06cd45d6e228827305ea5858ffc756984225dad4 Nov 02 22:10:45.789345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.789322 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b"] Nov 02 22:10:45.804124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.804104 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:45.862007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.861981 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62"] Nov 02 22:10:45.865309 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:45.865268 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ea6617_8b17_44c5_825c_2f5e4d80f388.slice/crio-19037d5dc04b790d49478224f29acf85bca1878be6fe75a2eecd5d43b73d3f36 WatchSource:0}: Error finding container 19037d5dc04b790d49478224f29acf85bca1878be6fe75a2eecd5d43b73d3f36: Status 404 returned error can't find the container with id 19037d5dc04b790d49478224f29acf85bca1878be6fe75a2eecd5d43b73d3f36 Nov 02 22:10:45.941985 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.941960 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-xks-ca\") pod \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " Nov 02 22:10:45.942179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.942168 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62f55ca6-8893-47ce-a6ed-c1357d1159ef-kube-api-access-mjrr2\" (UniqueName: \"kubernetes.io/projected/62f55ca6-8893-47ce-a6ed-c1357d1159ef-kube-api-access-mjrr2\") pod \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " Nov 02 22:10:45.942222 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.942205 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-openshift-ca\") pod \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\" (UID: \"62f55ca6-8893-47ce-a6ed-c1357d1159ef\") " Nov 02 22:10:45.942632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.942595 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-openshift-ca" pod "62f55ca6-8893-47ce-a6ed-c1357d1159ef" (UID: "62f55ca6-8893-47ce-a6ed-c1357d1159ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:45.982653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.982608 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-xks-ca" pod "62f55ca6-8893-47ce-a6ed-c1357d1159ef" (UID: "62f55ca6-8893-47ce-a6ed-c1357d1159ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:45.983338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:45.983302 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f55ca6-8893-47ce-a6ed-c1357d1159ef-kube-api-access-mjrr2" pod "62f55ca6-8893-47ce-a6ed-c1357d1159ef" (UID: "62f55ca6-8893-47ce-a6ed-c1357d1159ef"). InnerVolumeSpecName "kube-api-access-mjrr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:46.043476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.043449 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjrr2\" (UniqueName: \"kubernetes.io/projected/62f55ca6-8893-47ce-a6ed-c1357d1159ef-kube-api-access-mjrr2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:46.043476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.043475 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:46.043702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.043489 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62f55ca6-8893-47ce-a6ed-c1357d1159ef-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:46.079233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.079205 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj"] Nov 02 22:10:46.080060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.080042 4309 cpu_manager.go:414] "containerMap: removing container" podUID="62f55ca6-8893-47ce-a6ed-c1357d1159ef" containerName="revoke-keys" Nov 02 22:10:46.080149 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.080061 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62f55ca6-8893-47ce-a6ed-c1357d1159ef" containerName="revoke-keys" Nov 02 22:10:46.080190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.080182 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="62f55ca6-8893-47ce-a6ed-c1357d1159ef" containerName="revoke-keys" Nov 02 22:10:46.086944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.084953 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.090894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.090871 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj"] Nov 02 22:10:46.245919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.245886 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e806376-8ghhj\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.246033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.245945 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d754m\" (UniqueName: \"kubernetes.io/projected/24b6d07b-1016-49c7-a472-99f67b47f04b-kube-api-access-d754m\") pod \"maas-api-revoke-sub-3e806376-8ghhj\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.246218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.246191 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e806376-8ghhj\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.347538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.347346 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d754m\" (UniqueName: \"kubernetes.io/projected/24b6d07b-1016-49c7-a472-99f67b47f04b-kube-api-access-d754m\") pod \"maas-api-revoke-sub-3e806376-8ghhj\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.347729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.347545 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e806376-8ghhj\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.347729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.347608 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e806376-8ghhj\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.348249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.348225 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e806376-8ghhj\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.350323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.350297 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e806376-8ghhj\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.354995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.354965 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d754m\" (UniqueName: \"kubernetes.io/projected/24b6d07b-1016-49c7-a472-99f67b47f04b-kube-api-access-d754m\") pod \"maas-api-revoke-sub-3e806376-8ghhj\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.419102 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.419070 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:46.641218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.641183 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" event={"ID":"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d","Type":"ContainerStarted","Data":"ea3d153435f738db905e7870668495aa5170e986e21326316b57380cc2dd3c44"} Nov 02 22:10:46.641402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.641242 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" event={"ID":"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d","Type":"ContainerStarted","Data":"1e29895992d1e5d57f82b56746d205dd9fb081a89b0f3c96ad7f46111a4dc6bc"} Nov 02 22:10:46.642949 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.642927 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" Nov 02 22:10:46.643075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.642925 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f" event={"ID":"62f55ca6-8893-47ce-a6ed-c1357d1159ef","Type":"ContainerDied","Data":"42dfe6340756bd2db2c0ca55552d120e39159b5fab0120d2b1119edec5ef0ae8"} Nov 02 22:10:46.643075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.643044 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42dfe6340756bd2db2c0ca55552d120e39159b5fab0120d2b1119edec5ef0ae8" Nov 02 22:10:46.644712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.644659 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" event={"ID":"71ea6617-8b17-44c5-825c-2f5e4d80f388","Type":"ContainerStarted","Data":"6850c99fd259dbb63ae460d7bf8c443f5689e2185db8d20d98bf8efc6a06dbb4"} Nov 02 22:10:46.644817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.644731 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" event={"ID":"71ea6617-8b17-44c5-825c-2f5e4d80f388","Type":"ContainerStarted","Data":"19037d5dc04b790d49478224f29acf85bca1878be6fe75a2eecd5d43b73d3f36"} Nov 02 22:10:46.646969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.646819 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" event={"ID":"779db660-69dc-4cf4-a004-5b68ba24cdea","Type":"ContainerStarted","Data":"11719b3f9063767013525822659eb7f8168afacb19a34340df6d3c3b09bce369"} Nov 02 22:10:46.646969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.646866 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" event={"ID":"779db660-69dc-4cf4-a004-5b68ba24cdea","Type":"ContainerStarted","Data":"12bf90b49de01c2ee09394cc06cd45d6e228827305ea5858ffc756984225dad4"} Nov 02 22:10:46.656956 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.656904 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" podStartSLOduration=1.656888585 podStartE2EDuration="1.656888585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:46.655375927 +0000 UTC m=+774.593489144" watchObservedRunningTime="2026-11-02 22:10:46.656888585 +0000 UTC m=+774.595001803" Nov 02 22:10:46.675835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.675790 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" podStartSLOduration=1.6757769329999999 podStartE2EDuration="1.675776933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:46.672325939 +0000 UTC m=+774.610439157" watchObservedRunningTime="2026-11-02 22:10:46.675776933 +0000 UTC m=+774.613890151" Nov 02 22:10:46.690154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.690097 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" podStartSLOduration=1.690080013 podStartE2EDuration="1.690080013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:46.686031938 +0000 UTC m=+774.624145158" watchObservedRunningTime="2026-11-02 22:10:46.690080013 +0000 UTC m=+774.628193233" Nov 02 22:10:46.700722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:46.700463 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj"] Nov 02 22:10:46.702732 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:46.702706 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b6d07b_1016_49c7_a472_99f67b47f04b.slice/crio-d0716292812e80a5eba4457ded77f10d8b9a745c1462c947aaf6c63e88ccb3dc WatchSource:0}: Error finding container d0716292812e80a5eba4457ded77f10d8b9a745c1462c947aaf6c63e88ccb3dc: Status 404 returned error can't find the container with id d0716292812e80a5eba4457ded77f10d8b9a745c1462c947aaf6c63e88ccb3dc Nov 02 22:10:47.656632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:47.656548 4309 generic.go:356] "Generic (PLEG): container finished" podID="71ea6617-8b17-44c5-825c-2f5e4d80f388" containerID="6850c99fd259dbb63ae460d7bf8c443f5689e2185db8d20d98bf8efc6a06dbb4" exitCode=0 Nov 02 22:10:47.656632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:47.656596 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" event={"ID":"71ea6617-8b17-44c5-825c-2f5e4d80f388","Type":"ContainerDied","Data":"6850c99fd259dbb63ae460d7bf8c443f5689e2185db8d20d98bf8efc6a06dbb4"} Nov 02 22:10:47.658215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:47.658190 4309 generic.go:356] "Generic (PLEG): container finished" podID="779db660-69dc-4cf4-a004-5b68ba24cdea" containerID="11719b3f9063767013525822659eb7f8168afacb19a34340df6d3c3b09bce369" exitCode=0 Nov 02 22:10:47.658343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:47.658270 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" event={"ID":"779db660-69dc-4cf4-a004-5b68ba24cdea","Type":"ContainerDied","Data":"11719b3f9063767013525822659eb7f8168afacb19a34340df6d3c3b09bce369"} Nov 02 22:10:47.659749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:47.659726 4309 generic.go:356] "Generic (PLEG): container finished" podID="27ceaf50-e4ab-4ccb-b5b5-fad5e395669d" containerID="ea3d153435f738db905e7870668495aa5170e986e21326316b57380cc2dd3c44" exitCode=0 Nov 02 22:10:47.659866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:47.659808 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" event={"ID":"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d","Type":"ContainerDied","Data":"ea3d153435f738db905e7870668495aa5170e986e21326316b57380cc2dd3c44"} Nov 02 22:10:47.661374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:47.661352 4309 generic.go:356] "Generic (PLEG): container finished" podID="24b6d07b-1016-49c7-a472-99f67b47f04b" containerID="02b069cf87e256c2ef56b34c070dba9dc1482f5f98b37334f6b0a923c8a58374" exitCode=0 Nov 02 22:10:47.661469 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:47.661389 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" event={"ID":"24b6d07b-1016-49c7-a472-99f67b47f04b","Type":"ContainerDied","Data":"02b069cf87e256c2ef56b34c070dba9dc1482f5f98b37334f6b0a923c8a58374"} Nov 02 22:10:47.661469 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:47.661438 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" event={"ID":"24b6d07b-1016-49c7-a472-99f67b47f04b","Type":"ContainerStarted","Data":"d0716292812e80a5eba4457ded77f10d8b9a745c1462c947aaf6c63e88ccb3dc"} Nov 02 22:10:48.866709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.866669 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:48.975516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.975496 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-openshift-ca\") pod \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " Nov 02 22:10:48.975629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.975612 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-xks-ca\") pod \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " Nov 02 22:10:48.975795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.975778 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-kube-api-access-jmdsk\" (UniqueName: \"kubernetes.io/projected/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-kube-api-access-jmdsk\") pod \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\" (UID: \"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d\") " Nov 02 22:10:48.975897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.975872 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-openshift-ca" pod "27ceaf50-e4ab-4ccb-b5b5-fad5e395669d" (UID: "27ceaf50-e4ab-4ccb-b5b5-fad5e395669d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:48.976222 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.976205 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:48.977732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.977704 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-kube-api-access-jmdsk" pod "27ceaf50-e4ab-4ccb-b5b5-fad5e395669d" (UID: "27ceaf50-e4ab-4ccb-b5b5-fad5e395669d"). InnerVolumeSpecName "kube-api-access-jmdsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:48.977830 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.977756 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-xks-ca" pod "27ceaf50-e4ab-4ccb-b5b5-fad5e395669d" (UID: "27ceaf50-e4ab-4ccb-b5b5-fad5e395669d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:48.977881 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.977870 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:48.981433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.981415 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:48.987936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:48.987919 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:49.077127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.077094 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-xks-ca\") pod \"24b6d07b-1016-49c7-a472-99f67b47f04b\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " Nov 02 22:10:49.077295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.077195 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b6d07b-1016-49c7-a472-99f67b47f04b-kube-api-access-d754m\" (UniqueName: \"kubernetes.io/projected/24b6d07b-1016-49c7-a472-99f67b47f04b-kube-api-access-d754m\") pod \"24b6d07b-1016-49c7-a472-99f67b47f04b\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " Nov 02 22:10:49.077295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.077253 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-openshift-ca\") pod \"24b6d07b-1016-49c7-a472-99f67b47f04b\" (UID: \"24b6d07b-1016-49c7-a472-99f67b47f04b\") " Nov 02 22:10:49.077598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.077571 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmdsk\" (UniqueName: \"kubernetes.io/projected/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-kube-api-access-jmdsk\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.077598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.077588 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.078207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.078183 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-openshift-ca" pod "24b6d07b-1016-49c7-a472-99f67b47f04b" (UID: "24b6d07b-1016-49c7-a472-99f67b47f04b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:49.080305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.080284 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-xks-ca" pod "24b6d07b-1016-49c7-a472-99f67b47f04b" (UID: "24b6d07b-1016-49c7-a472-99f67b47f04b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:49.080405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.080328 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b6d07b-1016-49c7-a472-99f67b47f04b-kube-api-access-d754m" pod "24b6d07b-1016-49c7-a472-99f67b47f04b" (UID: "24b6d07b-1016-49c7-a472-99f67b47f04b"). InnerVolumeSpecName "kube-api-access-d754m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:49.178853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.178817 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-xks-ca\") pod \"779db660-69dc-4cf4-a004-5b68ba24cdea\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " Nov 02 22:10:49.179005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.178899 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-openshift-ca\") pod \"71ea6617-8b17-44c5-825c-2f5e4d80f388\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " Nov 02 22:10:49.179005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.178979 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779db660-69dc-4cf4-a004-5b68ba24cdea-kube-api-access-6j299\" (UniqueName: \"kubernetes.io/projected/779db660-69dc-4cf4-a004-5b68ba24cdea-kube-api-access-6j299\") pod \"779db660-69dc-4cf4-a004-5b68ba24cdea\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " Nov 02 22:10:49.179151 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179067 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-openshift-ca\") pod \"779db660-69dc-4cf4-a004-5b68ba24cdea\" (UID: \"779db660-69dc-4cf4-a004-5b68ba24cdea\") " Nov 02 22:10:49.179151 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179129 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-xks-ca\") pod \"71ea6617-8b17-44c5-825c-2f5e4d80f388\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " Nov 02 22:10:49.179289 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179264 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ea6617-8b17-44c5-825c-2f5e4d80f388-kube-api-access-txqzb\" (UniqueName: \"kubernetes.io/projected/71ea6617-8b17-44c5-825c-2f5e4d80f388-kube-api-access-txqzb\") pod \"71ea6617-8b17-44c5-825c-2f5e4d80f388\" (UID: \"71ea6617-8b17-44c5-825c-2f5e4d80f388\") " Nov 02 22:10:49.179412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179301 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-openshift-ca" pod "71ea6617-8b17-44c5-825c-2f5e4d80f388" (UID: "71ea6617-8b17-44c5-825c-2f5e4d80f388"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:49.179604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179577 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-openshift-ca" pod "779db660-69dc-4cf4-a004-5b68ba24cdea" (UID: "779db660-69dc-4cf4-a004-5b68ba24cdea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:49.179827 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179807 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.179914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179831 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.179914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179845 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d754m\" (UniqueName: \"kubernetes.io/projected/24b6d07b-1016-49c7-a472-99f67b47f04b-kube-api-access-d754m\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.179914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179858 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.179914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.179872 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b6d07b-1016-49c7-a472-99f67b47f04b-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.181179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.181154 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-xks-ca" pod "779db660-69dc-4cf4-a004-5b68ba24cdea" (UID: "779db660-69dc-4cf4-a004-5b68ba24cdea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:49.181430 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.181406 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ea6617-8b17-44c5-825c-2f5e4d80f388-kube-api-access-txqzb" pod "71ea6617-8b17-44c5-825c-2f5e4d80f388" (UID: "71ea6617-8b17-44c5-825c-2f5e4d80f388"). InnerVolumeSpecName "kube-api-access-txqzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:49.181765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.181745 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779db660-69dc-4cf4-a004-5b68ba24cdea-kube-api-access-6j299" pod "779db660-69dc-4cf4-a004-5b68ba24cdea" (UID: "779db660-69dc-4cf4-a004-5b68ba24cdea"). InnerVolumeSpecName "kube-api-access-6j299". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:49.181892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.181875 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-xks-ca" pod "71ea6617-8b17-44c5-825c-2f5e4d80f388" (UID: "71ea6617-8b17-44c5-825c-2f5e4d80f388"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:49.280647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.280578 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txqzb\" (UniqueName: \"kubernetes.io/projected/71ea6617-8b17-44c5-825c-2f5e4d80f388-kube-api-access-txqzb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.280647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.280605 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779db660-69dc-4cf4-a004-5b68ba24cdea-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.280647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.280616 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j299\" (UniqueName: \"kubernetes.io/projected/779db660-69dc-4cf4-a004-5b68ba24cdea-kube-api-access-6j299\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.280647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.280625 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ea6617-8b17-44c5-825c-2f5e4d80f388-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:49.676384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.676354 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" Nov 02 22:10:49.676384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.676361 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62" event={"ID":"71ea6617-8b17-44c5-825c-2f5e4d80f388","Type":"ContainerDied","Data":"19037d5dc04b790d49478224f29acf85bca1878be6fe75a2eecd5d43b73d3f36"} Nov 02 22:10:49.676602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.676413 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19037d5dc04b790d49478224f29acf85bca1878be6fe75a2eecd5d43b73d3f36" Nov 02 22:10:49.678071 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.677918 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" Nov 02 22:10:49.678071 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.677959 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b" event={"ID":"779db660-69dc-4cf4-a004-5b68ba24cdea","Type":"ContainerDied","Data":"12bf90b49de01c2ee09394cc06cd45d6e228827305ea5858ffc756984225dad4"} Nov 02 22:10:49.678071 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.678010 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12bf90b49de01c2ee09394cc06cd45d6e228827305ea5858ffc756984225dad4" Nov 02 22:10:49.679552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.679533 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" Nov 02 22:10:49.679707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.679571 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l" event={"ID":"27ceaf50-e4ab-4ccb-b5b5-fad5e395669d","Type":"ContainerDied","Data":"1e29895992d1e5d57f82b56746d205dd9fb081a89b0f3c96ad7f46111a4dc6bc"} Nov 02 22:10:49.679707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.679614 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e29895992d1e5d57f82b56746d205dd9fb081a89b0f3c96ad7f46111a4dc6bc" Nov 02 22:10:49.681155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.681133 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" event={"ID":"24b6d07b-1016-49c7-a472-99f67b47f04b","Type":"ContainerDied","Data":"d0716292812e80a5eba4457ded77f10d8b9a745c1462c947aaf6c63e88ccb3dc"} Nov 02 22:10:49.681226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.681175 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0716292812e80a5eba4457ded77f10d8b9a745c1462c947aaf6c63e88ccb3dc" Nov 02 22:10:49.681226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:49.681188 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj" Nov 02 22:10:51.731702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:51.731641 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:51.781034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:51.781001 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:52.918699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:52.899996 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:10:55.425942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.425904 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz"] Nov 02 22:10:55.426755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.426734 4309 cpu_manager.go:414] "containerMap: removing container" podUID="779db660-69dc-4cf4-a004-5b68ba24cdea" containerName="revoke-keys" Nov 02 22:10:55.426815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.426757 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779db660-69dc-4cf4-a004-5b68ba24cdea" containerName="revoke-keys" Nov 02 22:10:55.426815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.426783 4309 cpu_manager.go:414] "containerMap: removing container" podUID="71ea6617-8b17-44c5-825c-2f5e4d80f388" containerName="revoke-keys" Nov 02 22:10:55.426815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.426791 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ea6617-8b17-44c5-825c-2f5e4d80f388" containerName="revoke-keys" Nov 02 22:10:55.426815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.426806 4309 cpu_manager.go:414] "containerMap: removing container" podUID="24b6d07b-1016-49c7-a472-99f67b47f04b" containerName="revoke-keys" Nov 02 22:10:55.426938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.426815 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b6d07b-1016-49c7-a472-99f67b47f04b" containerName="revoke-keys" Nov 02 22:10:55.426938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.426828 4309 cpu_manager.go:414] "containerMap: removing container" podUID="27ceaf50-e4ab-4ccb-b5b5-fad5e395669d" containerName="revoke-keys" Nov 02 22:10:55.426938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.426836 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ceaf50-e4ab-4ccb-b5b5-fad5e395669d" containerName="revoke-keys" Nov 02 22:10:55.427042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.427029 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="779db660-69dc-4cf4-a004-5b68ba24cdea" containerName="revoke-keys" Nov 02 22:10:55.427076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.427049 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ea6617-8b17-44c5-825c-2f5e4d80f388" containerName="revoke-keys" Nov 02 22:10:55.427076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.427059 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b6d07b-1016-49c7-a472-99f67b47f04b" containerName="revoke-keys" Nov 02 22:10:55.427076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.427068 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ceaf50-e4ab-4ccb-b5b5-fad5e395669d" containerName="revoke-keys" Nov 02 22:10:55.435548 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.435521 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz"] Nov 02 22:10:55.435701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.435603 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.438956 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.438926 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:10:55.440047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.440026 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:10:55.440166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.440088 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:10:55.440166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.440106 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:10:55.499220 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.499186 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e96973-gs8pz\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.499370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.499334 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e96973-gs8pz\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.499453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.499436 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwgm\" (UniqueName: \"kubernetes.io/projected/2168bc7f-1970-46a2-9022-45b8704c2975-kube-api-access-rnwgm\") pod \"maas-api-revoke-sub-52e96973-gs8pz\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.600406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.600369 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwgm\" (UniqueName: \"kubernetes.io/projected/2168bc7f-1970-46a2-9022-45b8704c2975-kube-api-access-rnwgm\") pod \"maas-api-revoke-sub-52e96973-gs8pz\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.600592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.600472 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e96973-gs8pz\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.600592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.600580 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e96973-gs8pz\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.601197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.601178 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e96973-gs8pz\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.603445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.603420 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e96973-gs8pz\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.611232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.611205 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwgm\" (UniqueName: \"kubernetes.io/projected/2168bc7f-1970-46a2-9022-45b8704c2975-kube-api-access-rnwgm\") pod \"maas-api-revoke-sub-52e96973-gs8pz\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.671935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.671899 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f"] Nov 02 22:10:55.677087 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.677020 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.683975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.683946 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f"] Nov 02 22:10:55.684551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.684527 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a1d78c\"/\"distinct-w6-a1d78c-kserve-self-signed-certs\"" Nov 02 22:10:55.751853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.751806 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:55.807707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.804543 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-tmp-dir\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.807707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.804654 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a569f1-d1b9-4866-8f40-f8a6a398e239-tls-certs\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.807707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.804744 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-dshm\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.807707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.804787 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wp7d\" (UniqueName: \"kubernetes.io/projected/29a569f1-d1b9-4866-8f40-f8a6a398e239-kube-api-access-6wp7d\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.807707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.804820 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-home\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.807707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.804885 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-model-cache\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.906077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.906041 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-model-cache\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.906251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.906229 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-tmp-dir\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.906321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.906258 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a569f1-d1b9-4866-8f40-f8a6a398e239-tls-certs\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.906363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.906314 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-dshm\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.906363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.906354 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wp7d\" (UniqueName: \"kubernetes.io/projected/29a569f1-d1b9-4866-8f40-f8a6a398e239-kube-api-access-6wp7d\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.906433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.906383 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-home\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.906853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.906828 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-home\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.907078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.907059 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-model-cache\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.907304 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.907282 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-tmp-dir\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.910843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.910773 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a569f1-d1b9-4866-8f40-f8a6a398e239-tls-certs\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.910991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.910971 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-dshm\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.919556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.919531 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wp7d\" (UniqueName: \"kubernetes.io/projected/29a569f1-d1b9-4866-8f40-f8a6a398e239-kube-api-access-6wp7d\") pod \"distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:55.995266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:55.995183 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:56.081929 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:56.081903 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz"] Nov 02 22:10:56.283870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:56.283838 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f"] Nov 02 22:10:56.284383 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:56.284349 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29a569f1_d1b9_4866_8f40_f8a6a398e239.slice/crio-848417dce87c93a0f67e5248eeb92b44503a41b02a14be1f1a3790a6903cd82a WatchSource:0}: Error finding container 848417dce87c93a0f67e5248eeb92b44503a41b02a14be1f1a3790a6903cd82a: Status 404 returned error can't find the container with id 848417dce87c93a0f67e5248eeb92b44503a41b02a14be1f1a3790a6903cd82a Nov 02 22:10:56.763200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:56.763107 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" event={"ID":"2168bc7f-1970-46a2-9022-45b8704c2975","Type":"ContainerStarted","Data":"098cda27f9bacc98522a50a2dd778451520190993b8d7f0abf0fbca1a109528b"} Nov 02 22:10:56.763200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:56.763172 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" event={"ID":"2168bc7f-1970-46a2-9022-45b8704c2975","Type":"ContainerStarted","Data":"5cb37d9442815da61e17660b13308dafbb0aea9925b04d280f87f2d3e9e64229"} Nov 02 22:10:56.764996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:56.764968 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" event={"ID":"29a569f1-d1b9-4866-8f40-f8a6a398e239","Type":"ContainerStarted","Data":"e90610af491d0bd067b1790b5bb74e9514e7e2d83cb273efdf8df10b04e08c38"} Nov 02 22:10:56.765116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:56.765005 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" event={"ID":"29a569f1-d1b9-4866-8f40-f8a6a398e239","Type":"ContainerStarted","Data":"848417dce87c93a0f67e5248eeb92b44503a41b02a14be1f1a3790a6903cd82a"} Nov 02 22:10:56.779617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:56.779578 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" podStartSLOduration=1.779566264 podStartE2EDuration="1.779566264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:56.777383668 +0000 UTC m=+784.715496899" watchObservedRunningTime="2026-11-02 22:10:56.779566264 +0000 UTC m=+784.717679484" Nov 02 22:10:56.796872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:56.796832 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podStartSLOduration=1.796820037 podStartE2EDuration="1.796820037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:56.793825133 +0000 UTC m=+784.731938343" watchObservedRunningTime="2026-11-02 22:10:56.796820037 +0000 UTC m=+784.734933287" Nov 02 22:10:56.995627 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:56.995599 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:10:57.774493 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:57.774456 4309 generic.go:356] "Generic (PLEG): container finished" podID="2168bc7f-1970-46a2-9022-45b8704c2975" containerID="098cda27f9bacc98522a50a2dd778451520190993b8d7f0abf0fbca1a109528b" exitCode=0 Nov 02 22:10:57.774959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:57.774538 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" event={"ID":"2168bc7f-1970-46a2-9022-45b8704c2975","Type":"ContainerDied","Data":"098cda27f9bacc98522a50a2dd778451520190993b8d7f0abf0fbca1a109528b"} Nov 02 22:10:57.836190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:57.836158 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl"] Nov 02 22:10:57.840486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:57.840460 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:57.850144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:57.850104 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl"] Nov 02 22:10:57.930009 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:57.929969 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49131a6-gdqzl\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:57.930009 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:57.930012 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzb6\" (UniqueName: \"kubernetes.io/projected/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-kube-api-access-zjzb6\") pod \"maas-api-revoke-sub-f49131a6-gdqzl\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:57.930212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:57.930191 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49131a6-gdqzl\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:58.031380 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.031298 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49131a6-gdqzl\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:58.031380 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.031344 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzb6\" (UniqueName: \"kubernetes.io/projected/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-kube-api-access-zjzb6\") pod \"maas-api-revoke-sub-f49131a6-gdqzl\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:58.031569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.031444 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49131a6-gdqzl\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:58.032076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.032050 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f49131a6-gdqzl\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:58.034106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.034080 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f49131a6-gdqzl\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:58.039168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.039138 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzb6\" (UniqueName: \"kubernetes.io/projected/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-kube-api-access-zjzb6\") pod \"maas-api-revoke-sub-f49131a6-gdqzl\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:58.159820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.159786 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:10:58.437530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.437500 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl"] Nov 02 22:10:58.437650 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:10:58.437574 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c93d6cc_f250_4924_ab3a_d3f3a7855b93.slice/crio-1648796e83dba898fa7697628c3b5ae907dd6029451da9ef76a9183e0f9e94fa WatchSource:0}: Error finding container 1648796e83dba898fa7697628c3b5ae907dd6029451da9ef76a9183e0f9e94fa: Status 404 returned error can't find the container with id 1648796e83dba898fa7697628c3b5ae907dd6029451da9ef76a9183e0f9e94fa Nov 02 22:10:58.783322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.783281 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" event={"ID":"0c93d6cc-f250-4924-ab3a-d3f3a7855b93","Type":"ContainerStarted","Data":"0a252d0b7f457ea9a7dda2242e7cadaf8bebd870139ba426e8597c7bc2a1759c"} Nov 02 22:10:58.783855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.783347 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" event={"ID":"0c93d6cc-f250-4924-ab3a-d3f3a7855b93","Type":"ContainerStarted","Data":"1648796e83dba898fa7697628c3b5ae907dd6029451da9ef76a9183e0f9e94fa"} Nov 02 22:10:58.802644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.802586 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" podStartSLOduration=1.8025729529999999 podStartE2EDuration="1.802572953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:10:58.799100494 +0000 UTC m=+786.737213713" watchObservedRunningTime="2026-11-02 22:10:58.802572953 +0000 UTC m=+786.740686170" Nov 02 22:10:58.929878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:58.929840 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:59.044134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.044102 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-openshift-ca\") pod \"2168bc7f-1970-46a2-9022-45b8704c2975\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " Nov 02 22:10:59.044280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.044207 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-xks-ca\") pod \"2168bc7f-1970-46a2-9022-45b8704c2975\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " Nov 02 22:10:59.044280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.044248 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2168bc7f-1970-46a2-9022-45b8704c2975-kube-api-access-rnwgm\" (UniqueName: \"kubernetes.io/projected/2168bc7f-1970-46a2-9022-45b8704c2975-kube-api-access-rnwgm\") pod \"2168bc7f-1970-46a2-9022-45b8704c2975\" (UID: \"2168bc7f-1970-46a2-9022-45b8704c2975\") " Nov 02 22:10:59.044775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.044564 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-openshift-ca" pod "2168bc7f-1970-46a2-9022-45b8704c2975" (UID: "2168bc7f-1970-46a2-9022-45b8704c2975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:10:59.044909 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.044875 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:59.046514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.046492 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-xks-ca" pod "2168bc7f-1970-46a2-9022-45b8704c2975" (UID: "2168bc7f-1970-46a2-9022-45b8704c2975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:10:59.046888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.046871 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2168bc7f-1970-46a2-9022-45b8704c2975-kube-api-access-rnwgm" pod "2168bc7f-1970-46a2-9022-45b8704c2975" (UID: "2168bc7f-1970-46a2-9022-45b8704c2975"). InnerVolumeSpecName "kube-api-access-rnwgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:10:59.146443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.146408 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2168bc7f-1970-46a2-9022-45b8704c2975-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:59.146443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.146437 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnwgm\" (UniqueName: \"kubernetes.io/projected/2168bc7f-1970-46a2-9022-45b8704c2975-kube-api-access-rnwgm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:10:59.195581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.195552 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz"] Nov 02 22:10:59.196119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.196104 4309 cpu_manager.go:414] "containerMap: removing container" podUID="2168bc7f-1970-46a2-9022-45b8704c2975" containerName="revoke-keys" Nov 02 22:10:59.196119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.196119 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2168bc7f-1970-46a2-9022-45b8704c2975" containerName="revoke-keys" Nov 02 22:10:59.196259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.196242 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="2168bc7f-1970-46a2-9022-45b8704c2975" containerName="revoke-keys" Nov 02 22:10:59.199806 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.199786 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.209315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.209292 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz"] Nov 02 22:10:59.348261 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.348223 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85cb2fc2-qzkhz\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.348470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.348440 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/7dfed3e1-ca85-4586-ac5f-021853d85d90-kube-api-access-r8tsm\") pod \"maas-api-revoke-sub-85cb2fc2-qzkhz\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.348581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.348564 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85cb2fc2-qzkhz\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.411269 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.411234 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z"] Nov 02 22:10:59.415595 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.415571 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.430436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.430411 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z"] Nov 02 22:10:59.440757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.440729 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-361a44-openshift-default-dockercfg-6lxp8\"" Nov 02 22:10:59.449531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.449500 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85cb2fc2-qzkhz\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.449715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.449593 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85cb2fc2-qzkhz\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.449795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.449747 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/7dfed3e1-ca85-4586-ac5f-021853d85d90-kube-api-access-r8tsm\") pod \"maas-api-revoke-sub-85cb2fc2-qzkhz\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.450451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.450427 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85cb2fc2-qzkhz\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.452146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.452121 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85cb2fc2-qzkhz\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.457826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.457804 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/7dfed3e1-ca85-4586-ac5f-021853d85d90-kube-api-access-r8tsm\") pod \"maas-api-revoke-sub-85cb2fc2-qzkhz\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.517143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.517104 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:10:59.551307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.551242 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.551307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.551282 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.551452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.551354 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.551452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.551417 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.551522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.551478 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj29f\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-kube-api-access-jj29f\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.551556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.551534 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.551623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.551610 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.551660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.551631 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.551947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.551921 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.653604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.653568 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.653604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.653603 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.653925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.653654 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.653925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.653696 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.653925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.653713 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.653925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.653732 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.653925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.653805 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.653925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.653855 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj29f\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-kube-api-access-jj29f\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.653925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.653905 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.654351 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.654071 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.654351 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.654116 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.654479 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.654456 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.654549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.654524 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.654793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.654734 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.656118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.656094 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.656282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.656262 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.662592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.662564 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.662725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.662670 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj29f\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-kube-api-access-jj29f\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.695683 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.695637 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v"] Nov 02 22:10:59.702123 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.702096 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.715870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.715838 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v"] Nov 02 22:10:59.734561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.734527 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:10:59.755136 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.755110 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.755229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.755162 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.755277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.755240 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.755328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.755282 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.755328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.755313 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbeab10-9763-452d-93ca-a83d99de3652-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.755489 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.755474 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.755525 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.755507 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbeab10-9763-452d-93ca-a83d99de3652-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.755576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.755539 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h24qj\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-kube-api-access-h24qj\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.755757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.755737 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.795846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.795815 4309 generic.go:356] "Generic (PLEG): container finished" podID="0c93d6cc-f250-4924-ab3a-d3f3a7855b93" containerID="0a252d0b7f457ea9a7dda2242e7cadaf8bebd870139ba426e8597c7bc2a1759c" exitCode=0 Nov 02 22:10:59.796236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.795906 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" event={"ID":"0c93d6cc-f250-4924-ab3a-d3f3a7855b93","Type":"ContainerDied","Data":"0a252d0b7f457ea9a7dda2242e7cadaf8bebd870139ba426e8597c7bc2a1759c"} Nov 02 22:10:59.808639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.808604 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz"] Nov 02 22:10:59.810446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.810342 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" Nov 02 22:10:59.810654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.810579 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz" event={"ID":"2168bc7f-1970-46a2-9022-45b8704c2975","Type":"ContainerDied","Data":"5cb37d9442815da61e17660b13308dafbb0aea9925b04d280f87f2d3e9e64229"} Nov 02 22:10:59.810654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.810632 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb37d9442815da61e17660b13308dafbb0aea9925b04d280f87f2d3e9e64229" Nov 02 22:10:59.857080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.857041 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.857230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.857115 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.857230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.857149 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.857230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.857191 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.857390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.857233 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbeab10-9763-452d-93ca-a83d99de3652-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.857390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.857361 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.857487 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.857399 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbeab10-9763-452d-93ca-a83d99de3652-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.857487 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.857434 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h24qj\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-kube-api-access-h24qj\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.857592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.857503 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.858042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.858013 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.859427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.859396 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbeab10-9763-452d-93ca-a83d99de3652-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.859663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.859642 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.860192 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.860171 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.860335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.860310 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.860603 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.860575 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.861827 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.861809 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbeab10-9763-452d-93ca-a83d99de3652-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.870572 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.870547 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:10:59.871097 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:10:59.871076 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h24qj\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-kube-api-access-h24qj\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:11:00.017573 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.017535 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z"] Nov 02 22:11:00.017719 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:11:00.017532 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fc566d_a0b9_4072_964c_7ff08b445d4a.slice/crio-f27625b49b1469f6e8c7c74a5fbf994668ebee59a55d75e7ade0b5f5808c92ee WatchSource:0}: Error finding container f27625b49b1469f6e8c7c74a5fbf994668ebee59a55d75e7ade0b5f5808c92ee: Status 404 returned error can't find the container with id f27625b49b1469f6e8c7c74a5fbf994668ebee59a55d75e7ade0b5f5808c92ee Nov 02 22:11:00.020232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.020207 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:11:00.020490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.020461 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:00.020544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.020526 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:00.020580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.020555 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:00.321792 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.321765 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v"] Nov 02 22:11:00.322425 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:11:00.322397 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbeab10_9763_452d_93ca_a83d99de3652.slice/crio-332fda911c1cf38bd65e97e77e1a69356cb255421dbaf8124afeca9fc5e5213d WatchSource:0}: Error finding container 332fda911c1cf38bd65e97e77e1a69356cb255421dbaf8124afeca9fc5e5213d: Status 404 returned error can't find the container with id 332fda911c1cf38bd65e97e77e1a69356cb255421dbaf8124afeca9fc5e5213d Nov 02 22:11:00.325441 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.325405 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:00.325538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.325466 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:00.325538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.325498 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:00.836107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.836023 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" event={"ID":"7dfed3e1-ca85-4586-ac5f-021853d85d90","Type":"ContainerStarted","Data":"1ef600122737efda3914947957b4669d2edcb3bf1fdaac4a97080cb29954a576"} Nov 02 22:11:00.836107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.836077 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" event={"ID":"7dfed3e1-ca85-4586-ac5f-021853d85d90","Type":"ContainerStarted","Data":"8cf1e99382d08ebb4bd2ad01b0637dd6c501f633e787509e6b36b492920274eb"} Nov 02 22:11:00.838410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.838382 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" event={"ID":"bdbeab10-9763-452d-93ca-a83d99de3652","Type":"ContainerStarted","Data":"a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007"} Nov 02 22:11:00.840482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.840463 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" event={"ID":"bdbeab10-9763-452d-93ca-a83d99de3652","Type":"ContainerStarted","Data":"332fda911c1cf38bd65e97e77e1a69356cb255421dbaf8124afeca9fc5e5213d"} Nov 02 22:11:00.840969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.840949 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" event={"ID":"b1fc566d-a0b9-4072-964c-7ff08b445d4a","Type":"ContainerStarted","Data":"bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5"} Nov 02 22:11:00.841045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.840996 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" event={"ID":"b1fc566d-a0b9-4072-964c-7ff08b445d4a","Type":"ContainerStarted","Data":"f27625b49b1469f6e8c7c74a5fbf994668ebee59a55d75e7ade0b5f5808c92ee"} Nov 02 22:11:00.859701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.859622 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" podStartSLOduration=1.859601494 podStartE2EDuration="1.859601494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:00.85419576 +0000 UTC m=+788.792308992" watchObservedRunningTime="2026-11-02 22:11:00.859601494 +0000 UTC m=+788.797714714" Nov 02 22:11:00.918896 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.918831 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" podStartSLOduration=1.9188064759999999 podStartE2EDuration="1.918806476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:00.916628423 +0000 UTC m=+788.854741650" watchObservedRunningTime="2026-11-02 22:11:00.918806476 +0000 UTC m=+788.856919697" Nov 02 22:11:00.961600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:00.961525 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podStartSLOduration=1.961501718 podStartE2EDuration="1.961501718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:00.948950943 +0000 UTC m=+788.887064163" watchObservedRunningTime="2026-11-02 22:11:00.961501718 +0000 UTC m=+788.899614937" Nov 02 22:11:01.054724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.054656 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:11:01.059669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.059314 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:11:01.059669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.059378 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:11:01.178156 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.178129 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:11:01.346947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.346919 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-kube-api-access-zjzb6\" (UniqueName: \"kubernetes.io/projected/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-kube-api-access-zjzb6\") pod \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " Nov 02 22:11:01.348131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.347269 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-openshift-ca\") pod \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " Nov 02 22:11:01.348131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.347337 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-xks-ca\") pod \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\" (UID: \"0c93d6cc-f250-4924-ab3a-d3f3a7855b93\") " Nov 02 22:11:01.348131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.347970 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-openshift-ca" pod "0c93d6cc-f250-4924-ab3a-d3f3a7855b93" (UID: "0c93d6cc-f250-4924-ab3a-d3f3a7855b93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:01.350708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.349872 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-xks-ca" pod "0c93d6cc-f250-4924-ab3a-d3f3a7855b93" (UID: "0c93d6cc-f250-4924-ab3a-d3f3a7855b93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:01.350708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.349948 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-kube-api-access-zjzb6" pod "0c93d6cc-f250-4924-ab3a-d3f3a7855b93" (UID: "0c93d6cc-f250-4924-ab3a-d3f3a7855b93"). InnerVolumeSpecName "kube-api-access-zjzb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:01.449467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.449424 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:01.449467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.449463 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:01.449467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.449474 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjzb6\" (UniqueName: \"kubernetes.io/projected/0c93d6cc-f250-4924-ab3a-d3f3a7855b93-kube-api-access-zjzb6\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:01.737748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.736843 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:11:01.737748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.737017 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 22:11:01.737748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.737075 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 22:11:01.850535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.850496 4309 generic.go:356] "Generic (PLEG): container finished" podID="7dfed3e1-ca85-4586-ac5f-021853d85d90" containerID="1ef600122737efda3914947957b4669d2edcb3bf1fdaac4a97080cb29954a576" exitCode=0 Nov 02 22:11:01.851019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.850627 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" event={"ID":"7dfed3e1-ca85-4586-ac5f-021853d85d90","Type":"ContainerDied","Data":"1ef600122737efda3914947957b4669d2edcb3bf1fdaac4a97080cb29954a576"} Nov 02 22:11:01.853184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.853164 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" Nov 02 22:11:01.853380 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.853185 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl" event={"ID":"0c93d6cc-f250-4924-ab3a-d3f3a7855b93","Type":"ContainerDied","Data":"1648796e83dba898fa7697628c3b5ae907dd6029451da9ef76a9183e0f9e94fa"} Nov 02 22:11:01.853588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:01.853575 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1648796e83dba898fa7697628c3b5ae907dd6029451da9ef76a9183e0f9e94fa" Nov 02 22:11:02.024528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:02.024397 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:11:02.024528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:02.024468 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:11:02.737734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:02.736146 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 22:11:02.737734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:02.736211 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 22:11:03.052275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.052090 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:11:03.053310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.052170 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:11:03.466772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.463822 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:11:03.551704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.548063 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-xks-ca\") pod \"7dfed3e1-ca85-4586-ac5f-021853d85d90\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " Nov 02 22:11:03.551704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.548159 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-openshift-ca\") pod \"7dfed3e1-ca85-4586-ac5f-021853d85d90\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " Nov 02 22:11:03.551704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.548318 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfed3e1-ca85-4586-ac5f-021853d85d90-kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/7dfed3e1-ca85-4586-ac5f-021853d85d90-kube-api-access-r8tsm\") pod \"7dfed3e1-ca85-4586-ac5f-021853d85d90\" (UID: \"7dfed3e1-ca85-4586-ac5f-021853d85d90\") " Nov 02 22:11:03.551704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.550314 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-openshift-ca" pod "7dfed3e1-ca85-4586-ac5f-021853d85d90" (UID: "7dfed3e1-ca85-4586-ac5f-021853d85d90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:03.554695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.554639 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfed3e1-ca85-4586-ac5f-021853d85d90-kube-api-access-r8tsm" pod "7dfed3e1-ca85-4586-ac5f-021853d85d90" (UID: "7dfed3e1-ca85-4586-ac5f-021853d85d90"). InnerVolumeSpecName "kube-api-access-r8tsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:03.555886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.555869 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-xks-ca" pod "7dfed3e1-ca85-4586-ac5f-021853d85d90" (UID: "7dfed3e1-ca85-4586-ac5f-021853d85d90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:03.652701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.651651 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:03.652701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.651713 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dfed3e1-ca85-4586-ac5f-021853d85d90-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:03.652701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.651730 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8tsm\" (UniqueName: \"kubernetes.io/projected/7dfed3e1-ca85-4586-ac5f-021853d85d90-kube-api-access-r8tsm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:03.736757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.736117 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 22:11:03.736757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.736179 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 22:11:03.868663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.868622 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" event={"ID":"7dfed3e1-ca85-4586-ac5f-021853d85d90","Type":"ContainerDied","Data":"8cf1e99382d08ebb4bd2ad01b0637dd6c501f633e787509e6b36b492920274eb"} Nov 02 22:11:03.868842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.868711 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cf1e99382d08ebb4bd2ad01b0637dd6c501f633e787509e6b36b492920274eb" Nov 02 22:11:03.868842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:03.868766 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz" Nov 02 22:11:04.023776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.023586 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:11:04.023776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.023646 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:11:04.444697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.427601 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn"] Nov 02 22:11:04.446174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.446145 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7dfed3e1-ca85-4586-ac5f-021853d85d90" containerName="revoke-keys" Nov 02 22:11:04.446323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.446309 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfed3e1-ca85-4586-ac5f-021853d85d90" containerName="revoke-keys" Nov 02 22:11:04.446509 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.446496 4309 cpu_manager.go:414] "containerMap: removing container" podUID="0c93d6cc-f250-4924-ab3a-d3f3a7855b93" containerName="revoke-keys" Nov 02 22:11:04.446603 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.446591 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c93d6cc-f250-4924-ab3a-d3f3a7855b93" containerName="revoke-keys" Nov 02 22:11:04.446858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.446842 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c93d6cc-f250-4924-ab3a-d3f3a7855b93" containerName="revoke-keys" Nov 02 22:11:04.447011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.446999 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfed3e1-ca85-4586-ac5f-021853d85d90" containerName="revoke-keys" Nov 02 22:11:04.470015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.465719 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng"] Nov 02 22:11:04.470015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.466539 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.471221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.471199 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn"] Nov 02 22:11:04.471495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.471432 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.473211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.473194 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng"] Nov 02 22:11:04.562861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.562820 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.562861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.562862 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.563126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.562939 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.563126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.562991 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.563126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563015 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.563126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563033 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.563126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563066 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.563126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563087 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.563126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563103 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxwq\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-kube-api-access-tpxwq\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.563506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563144 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.563506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563189 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.563506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563211 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.563506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563251 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.563506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563296 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.563506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563321 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.563506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563342 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.563506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563356 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3a00cf-3743-4d16-932d-99c767e58d83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.563506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.563372 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q26nw\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-kube-api-access-q26nw\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.664312 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664278 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.664312 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664317 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.664515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664337 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxwq\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-kube-api-access-tpxwq\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.664515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664391 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.664515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664454 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.664515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664486 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.664642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664544 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.664642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664611 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.664755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664651 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.664755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664702 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.664755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664727 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3a00cf-3743-4d16-932d-99c767e58d83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.664755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664750 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q26nw\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-kube-api-access-q26nw\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.664968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664785 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.664968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664810 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.664968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664835 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.664968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664877 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.664968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664896 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.664968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.664912 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.666637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.665492 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.666637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.665617 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.666637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.665878 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.666637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.666143 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3a00cf-3743-4d16-932d-99c767e58d83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.666637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.666175 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.666637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.666399 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.666637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.666434 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.666637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.666577 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.667171 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.666725 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.667453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.667340 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.667453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.667415 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.667805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.667784 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.667805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.667790 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.668825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.668805 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.674791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.674772 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.675502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.675485 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.676001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.675983 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q26nw\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-kube-api-access-q26nw\") pod \"maas-default-gateway-openshift-default-696f47c6c-8vgmn\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:04.676246 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.676225 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxwq\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-kube-api-access-tpxwq\") pod \"maas-default-gateway-openshift-default-696f47c6c-r5hng\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.740774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.736132 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 22:11:04.740774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.736196 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 22:11:04.857958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.857922 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:04.858545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:04.858524 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:05.024121 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.022631 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:11:05.024121 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.022700 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:11:05.277184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.277153 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng"] Nov 02 22:11:05.280448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.280409 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:05.280526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.280492 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:05.280571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.280530 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:05.675881 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.669574 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn"] Nov 02 22:11:05.734840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.734797 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:05.735079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.735056 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:05.735216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.735198 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:05.747793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.747765 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 22:11:05.748009 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.747972 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 22:11:05.928535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.928497 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" event={"ID":"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2","Type":"ContainerStarted","Data":"660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4"} Nov 02 22:11:05.928832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.928809 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" event={"ID":"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2","Type":"ContainerStarted","Data":"fa2b00d7ca1824538a0f74cededa3578e3bd736329f01f0844325b653d66c7f0"} Nov 02 22:11:05.944217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.944187 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" event={"ID":"4e3a00cf-3743-4d16-932d-99c767e58d83","Type":"ContainerStarted","Data":"ec1ebe848c21708db5afcf3bd8b973bfc74a58d7aa66cdf39dfd03a623dfd78f"} Nov 02 22:11:05.960431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:05.960377 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podStartSLOduration=1.9603600939999999 podStartE2EDuration="1.960360094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:05.958156012 +0000 UTC m=+793.896269231" watchObservedRunningTime="2026-11-02 22:11:05.960360094 +0000 UTC m=+793.898473318" Nov 02 22:11:06.009700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.008002 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:11:06.038703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.035251 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:11:06.038954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.038922 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:11:06.074445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.073498 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:11:06.735699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.735234 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 22:11:06.735699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.735295 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 22:11:06.859344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.859309 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:06.860937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.860911 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:11:06.861085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.860962 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:11:06.964824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:06.964785 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" event={"ID":"4e3a00cf-3743-4d16-932d-99c767e58d83","Type":"ContainerStarted","Data":"e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e"} Nov 02 22:11:07.063054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.062954 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:11:07.111701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.103386 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podStartSLOduration=3.10336412 podStartE2EDuration="3.10336412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:07.00627038 +0000 UTC m=+794.944383598" watchObservedRunningTime="2026-11-02 22:11:07.10336412 +0000 UTC m=+795.041477339" Nov 02 22:11:07.198803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.186225 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 22:11:07.198803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.186301 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 22:11:07.758697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.758531 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:11:07.759224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.759191 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:11:07.760267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.760246 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:11:07.863164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.863127 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:11:07.863454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.863415 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:11:07.865346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.865324 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:07.865602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.865583 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:11:07.865767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:07.865745 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:11:08.051118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:08.051029 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:11:08.052855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:08.052834 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:11:08.053906 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:08.053884 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:11:08.864111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:08.864069 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:11:08.864747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:08.864704 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:11:08.865599 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:08.865576 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:11:08.865790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:08.865766 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:11:09.619605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.619573 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph"] Nov 02 22:11:09.739921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.739890 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph"] Nov 02 22:11:09.762404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.740218 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:09.763038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.762970 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:09.799950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.799889 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:11:09.800394 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.800365 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:11:09.908082 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.907971 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:09.909034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.908993 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:11:09.909218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.909041 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:11:09.963699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.909338 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:11:09.963699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.955826 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:11:09.963699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.952242 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a279d67-bbzph\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:09.963699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.956011 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a279d67-bbzph\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:09.963699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.956141 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9xz\" (UniqueName: \"kubernetes.io/projected/0dd30d87-804c-4620-8d90-a9448b1d8437-kube-api-access-nq9xz\") pod \"maas-api-revoke-sub-8a279d67-bbzph\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:09.963699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.936024 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:09.963699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.910568 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:11:09.963699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.956643 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:11:09.985335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:09.983862 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:10.102698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.065624 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a279d67-bbzph\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:10.102698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.065826 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a279d67-bbzph\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:10.102698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.065952 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9xz\" (UniqueName: \"kubernetes.io/projected/0dd30d87-804c-4620-8d90-a9448b1d8437-kube-api-access-nq9xz\") pod \"maas-api-revoke-sub-8a279d67-bbzph\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:10.102698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.066570 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a279d67-bbzph\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:10.133706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.131884 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9xz\" (UniqueName: \"kubernetes.io/projected/0dd30d87-804c-4620-8d90-a9448b1d8437-kube-api-access-nq9xz\") pod \"maas-api-revoke-sub-8a279d67-bbzph\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:10.148212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.148176 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a279d67-bbzph\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:10.185765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.177491 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Nov 02 22:11:10.389700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.388001 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:10.623490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.622696 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:11:10.765765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.764815 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:10.765958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.765806 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:11:10.929245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.929197 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:11:10.929795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.929746 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:11:10.969462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.961518 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:11:10.969462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.961592 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:11:10.969462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.968783 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:11.004700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.970344 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:11:11.036754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.970823 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:11.072456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:10.997165 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:11.073062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.073040 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:11:11.073239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.073226 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:11:11.625790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.625743 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:11.633027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.632464 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:11:11.812222 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.812169 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:11:11.941127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.878744 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:11:11.941127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.921112 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:11:11.941127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.886609 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:11:11.941127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.921182 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:11:11.941127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.902263 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:11.987697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:11.980375 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:12.022188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.007459 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:12.125704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.125328 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:11:12.144809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.126898 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:11:12.303215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.303182 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:11:12.303499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.303481 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:11:12.303630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.303616 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph"] Nov 02 22:11:12.391646 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:11:12.391595 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:11:12.416288 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.416177 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:11:12.570297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.570267 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:11:12.573155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.573128 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:11:12.669796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.669712 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" event={"ID":"0dd30d87-804c-4620-8d90-a9448b1d8437","Type":"ContainerStarted","Data":"a7dec54052f99d0137b4db8500f8608f9dca637f10904c9a6c2f5ae705f2e83c"} Nov 02 22:11:12.860117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.860078 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:11:12.860307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.860282 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:11:12.860724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.860702 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:11:12.860891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:12.860870 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:11:13.709699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:13.709533 4309 generic.go:356] "Generic (PLEG): container finished" podID="0dd30d87-804c-4620-8d90-a9448b1d8437" containerID="88097bdafedd8b996b829a16f9679ab4f7f5445afe8b3b763bcb2dfb02c97b44" exitCode=0 Nov 02 22:11:13.710202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:13.709807 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" event={"ID":"0dd30d87-804c-4620-8d90-a9448b1d8437","Type":"ContainerDied","Data":"88097bdafedd8b996b829a16f9679ab4f7f5445afe8b3b763bcb2dfb02c97b44"} Nov 02 22:11:13.859732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:13.859621 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:11:13.859732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:13.859621 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:11:13.859732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:13.859713 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:11:13.859732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:13.859728 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:11:14.433258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.433220 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6"] Nov 02 22:11:14.437609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.437592 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.450222 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.450195 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6"] Nov 02 22:11:14.479247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.479209 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.479417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.479266 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhw7b\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-kube-api-access-zhw7b\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.479417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.479291 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.479417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.479401 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.479605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.479466 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.479605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.479513 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.479773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.479651 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.479773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.479715 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.479892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.479787 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.581669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.581636 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhw7b\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-kube-api-access-zhw7b\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.582403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.581944 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.582616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.582597 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.582820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.582794 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.582948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.582849 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.583017 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.582992 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.588670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.584092 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.588670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.586963 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.591838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.591217 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.591838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.591633 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.594540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.594481 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhw7b\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-kube-api-access-zhw7b\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.603380 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.599896 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85"] Nov 02 22:11:14.616119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.601484 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.616119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.603695 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.616119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.603764 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.616119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.604474 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.616119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.604746 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.623693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.621046 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.642372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.642319 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-r27p6\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.645881 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.645858 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85"] Nov 02 22:11:14.646062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.645991 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.704761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.704637 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.705070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.705048 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.705209 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.705194 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.705322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.705308 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.705470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.705456 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.705585 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.705569 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.705717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.705702 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf4l2\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-kube-api-access-vf4l2\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.705865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.705850 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.705988 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.705975 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.760721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.760654 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:14.821120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.821087 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.823716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.821503 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.823716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.821584 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.823716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.821634 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.823716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.821661 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.823716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.821713 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.825125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.824254 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.825125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.824305 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.825125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.824353 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf4l2\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-kube-api-access-vf4l2\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.825125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.823285 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.825125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.824146 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.825125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.822949 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.825608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.823181 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.826013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.823044 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.856664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.836824 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.856664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.839894 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.856664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.840381 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.856664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.846855 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf4l2\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-kube-api-access-vf4l2\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:14.874260 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.874224 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:11:14.874432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.874284 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:11:14.874432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.874238 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 02 22:11:14.874432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.874340 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 02 22:11:14.939435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.939398 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:14.967085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:14.967007 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:15.055955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.034137 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd30d87-804c-4620-8d90-a9448b1d8437-kube-api-access-nq9xz\" (UniqueName: \"kubernetes.io/projected/0dd30d87-804c-4620-8d90-a9448b1d8437-kube-api-access-nq9xz\") pod \"0dd30d87-804c-4620-8d90-a9448b1d8437\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " Nov 02 22:11:15.055955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.034394 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-xks-ca\") pod \"0dd30d87-804c-4620-8d90-a9448b1d8437\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " Nov 02 22:11:15.055955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.034503 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-openshift-ca\") pod \"0dd30d87-804c-4620-8d90-a9448b1d8437\" (UID: \"0dd30d87-804c-4620-8d90-a9448b1d8437\") " Nov 02 22:11:15.055955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.037660 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-openshift-ca" pod "0dd30d87-804c-4620-8d90-a9448b1d8437" (UID: "0dd30d87-804c-4620-8d90-a9448b1d8437"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:15.079460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.079149 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd30d87-804c-4620-8d90-a9448b1d8437-kube-api-access-nq9xz" pod "0dd30d87-804c-4620-8d90-a9448b1d8437" (UID: "0dd30d87-804c-4620-8d90-a9448b1d8437"). InnerVolumeSpecName "kube-api-access-nq9xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:15.110855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.110792 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-xks-ca" pod "0dd30d87-804c-4620-8d90-a9448b1d8437" (UID: "0dd30d87-804c-4620-8d90-a9448b1d8437"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:15.149963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.149929 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq9xz\" (UniqueName: \"kubernetes.io/projected/0dd30d87-804c-4620-8d90-a9448b1d8437-kube-api-access-nq9xz\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:15.150081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.149970 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:15.150081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.149987 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd30d87-804c-4620-8d90-a9448b1d8437-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:15.153301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.150260 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6"] Nov 02 22:11:15.156498 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.156459 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:15.156767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.156649 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:15.157063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.156888 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:15.395363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.394607 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:15.395363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.394719 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:15.395363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.394766 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:15.397661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.397585 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85"] Nov 02 22:11:15.738611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.738562 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" event={"ID":"52b21749-629f-416a-9ad1-5ab65ec9ccd8","Type":"ContainerStarted","Data":"6931dbf3e31ced6b76061acbd6dd269bb50f77847e50eefa4b0d8db81f926ba6"} Nov 02 22:11:15.739089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.738657 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" event={"ID":"52b21749-629f-416a-9ad1-5ab65ec9ccd8","Type":"ContainerStarted","Data":"d231a13cdb47b615f700b8c7424beff06f7a3fe41deb2ac1fb3d8f10bbb5a7dc"} Nov 02 22:11:15.740805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.740780 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" Nov 02 22:11:15.740927 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.740853 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph" event={"ID":"0dd30d87-804c-4620-8d90-a9448b1d8437","Type":"ContainerDied","Data":"a7dec54052f99d0137b4db8500f8608f9dca637f10904c9a6c2f5ae705f2e83c"} Nov 02 22:11:15.740927 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.740897 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7dec54052f99d0137b4db8500f8608f9dca637f10904c9a6c2f5ae705f2e83c" Nov 02 22:11:15.743473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.743447 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" event={"ID":"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2","Type":"ContainerStarted","Data":"d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3"} Nov 02 22:11:15.743638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.743618 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" event={"ID":"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2","Type":"ContainerStarted","Data":"47eabb83794617f867f6522eaf8a47e1f65a0901aeda296bc90036ddf5264ce1"} Nov 02 22:11:15.761137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.761117 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:15.782782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.782613 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" podStartSLOduration=1.78259406 podStartE2EDuration="1.78259406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:15.774727356 +0000 UTC m=+803.712840576" watchObservedRunningTime="2026-11-02 22:11:15.78259406 +0000 UTC m=+803.720707278" Nov 02 22:11:15.801516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.801445 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" podStartSLOduration=1.8014256469999999 podStartE2EDuration="1.801425647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:15.794307645 +0000 UTC m=+803.732420893" watchObservedRunningTime="2026-11-02 22:11:15.801425647 +0000 UTC m=+803.739538869" Nov 02 22:11:15.874700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.869324 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 22:11:15.874700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.869378 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 22:11:15.967977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:15.967932 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:16.774931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:16.774892 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:11:16.775557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:16.775526 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:11:16.905919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:16.905877 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:11:16.906209 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:16.906177 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:16.969843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:16.969803 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:11:16.970114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:16.970085 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:11:17.531700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:17.531019 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:17.535695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:17.532420 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:17.538490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:17.538469 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:11:17.790252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:17.768504 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:11:17.805862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:17.805819 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:11:17.908389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:17.907792 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:11:17.908389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:17.907869 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:17.990125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:17.990078 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:11:17.990422 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:17.990377 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:11:18.413332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:18.413295 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:18.446705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:18.436228 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:18.449939 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:18.449904 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:11:18.762534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:18.762423 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:11:18.762534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:18.762497 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:11:18.968213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:18.968180 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:11:18.968592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:18.968234 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:11:19.761133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:19.761100 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:11:19.761321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:19.761162 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:11:19.968214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:19.968182 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:11:19.968457 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:19.968429 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:11:20.761729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:20.761695 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:11:20.761905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:20.761748 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:11:20.967320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:20.967290 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:11:20.967466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:20.967339 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:11:21.761658 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:21.761622 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 02 22:11:21.762069 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:21.761725 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 02 22:11:21.968098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:21.968058 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 22:11:21.968321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:21.968118 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 22:11:22.765299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.765267 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:22.765799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.765708 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:22.766444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.766426 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:11:22.927044 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.927011 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l"] Nov 02 22:11:22.929839 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.929810 4309 cpu_manager.go:414] "containerMap: removing container" podUID="0dd30d87-804c-4620-8d90-a9448b1d8437" containerName="revoke-keys" Nov 02 22:11:22.929839 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.929838 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd30d87-804c-4620-8d90-a9448b1d8437" containerName="revoke-keys" Nov 02 22:11:22.930007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.929980 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd30d87-804c-4620-8d90-a9448b1d8437" containerName="revoke-keys" Nov 02 22:11:22.936001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.935976 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:22.939447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.939420 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l"] Nov 02 22:11:22.946014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.945845 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-310bb3\"/\"distinct-w0-310bb3-kserve-self-signed-certs\"" Nov 02 22:11:22.973491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.973463 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:22.973994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.973973 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:22.974743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.974722 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:11:22.999748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.999715 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tmp-dir\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:22.999938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.999820 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-dshm\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:22.999938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.999904 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tls-certs\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.000054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.999942 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-home\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.000054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:22.999961 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-model-cache\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.000162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.000061 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99qhb\" (UniqueName: \"kubernetes.io/projected/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-kube-api-access-99qhb\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.101934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.101894 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tmp-dir\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.102282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.102254 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-dshm\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.102417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.102366 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tls-certs\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.102483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.102418 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-home\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.102483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.102448 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-model-cache\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.102483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.102455 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tmp-dir\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.102652 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.102567 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99qhb\" (UniqueName: \"kubernetes.io/projected/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-kube-api-access-99qhb\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.102760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.102742 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-model-cache\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.102873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.102832 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-home\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.104861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.104833 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-dshm\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.105038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.105016 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tls-certs\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.111436 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99qhb\" (UniqueName: \"kubernetes.io/projected/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-kube-api-access-99qhb\") pod \"distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.252649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.252611 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:23.554556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.554512 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l"] Nov 02 22:11:23.555719 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:11:23.555659 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd18a5ea4_60d8_43eb_a7c2_b43f946d0a9a.slice/crio-a2c4b4e76bbb12a54db0d5e7ff4e7c124f770bb9c27a5f1f9576157b4e10c275 WatchSource:0}: Error finding container a2c4b4e76bbb12a54db0d5e7ff4e7c124f770bb9c27a5f1f9576157b4e10c275: Status 404 returned error can't find the container with id a2c4b4e76bbb12a54db0d5e7ff4e7c124f770bb9c27a5f1f9576157b4e10c275 Nov 02 22:11:23.980864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.980826 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" event={"ID":"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a","Type":"ContainerStarted","Data":"0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba"} Nov 02 22:11:23.981270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.980875 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" event={"ID":"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a","Type":"ContainerStarted","Data":"a2c4b4e76bbb12a54db0d5e7ff4e7c124f770bb9c27a5f1f9576157b4e10c275"} Nov 02 22:11:23.997087 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:23.997029 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podStartSLOduration=1.997011633 podStartE2EDuration="1.997011633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:23.995797711 +0000 UTC m=+811.933910932" watchObservedRunningTime="2026-11-02 22:11:23.997011633 +0000 UTC m=+811.935124851" Nov 02 22:11:24.253431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:24.253352 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:27.846702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:27.839766 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:28.135779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.131318 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:28.409775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.408752 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2"] Nov 02 22:11:28.592721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.592620 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:11:28.616610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.593093 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 02 22:11:28.616610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.593904 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.664094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.663632 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c302b6\"/\"openshift-service-ca.crt\"" Nov 02 22:11:28.664094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.663760 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c302b6\"/\"kube-root-ca.crt\"" Nov 02 22:11:28.664094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.663902 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c302b6\"/\"default-dockercfg-p55xm\"" Nov 02 22:11:28.665561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.665425 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c302b6\"/\"distinct-w4-c302b6-kserve-self-signed-certs\"" Nov 02 22:11:28.761828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.761797 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2"] Nov 02 22:11:28.765832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.765292 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-tmp-dir\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.766006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.765981 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv765\" (UniqueName: \"kubernetes.io/projected/edf36169-2951-4b4a-a8bc-20b19196adaf-kube-api-access-pv765\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.766078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.766054 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-home\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.766140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.766113 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-dshm\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.766193 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.766139 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf36169-2951-4b4a-a8bc-20b19196adaf-tls-certs\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.766193 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.766179 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-model-cache\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.872714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.867334 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-model-cache\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.872714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.867914 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-model-cache\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.872714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.868112 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-tmp-dir\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.872714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.868274 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv765\" (UniqueName: \"kubernetes.io/projected/edf36169-2951-4b4a-a8bc-20b19196adaf-kube-api-access-pv765\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.872714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.868348 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-home\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.872714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.868403 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-dshm\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.872714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.868455 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf36169-2951-4b4a-a8bc-20b19196adaf-tls-certs\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.872714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.869476 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-tmp-dir\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.872714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.869713 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-home\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.875436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.875412 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf36169-2951-4b4a-a8bc-20b19196adaf-tls-certs\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.877224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.877200 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-dshm\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:28.877565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:28.877543 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv765\" (UniqueName: \"kubernetes.io/projected/edf36169-2951-4b4a-a8bc-20b19196adaf-kube-api-access-pv765\") pod \"distinct-w4-c302b6-kserve-775b8f776f-px9c2\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:29.064607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.064584 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:29.230208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.230176 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj"] Nov 02 22:11:29.235965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.235942 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.251634 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.251599 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj"] Nov 02 22:11:29.274038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.273977 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.274038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.274033 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.274308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.274082 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.274308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.274117 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.274308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.274240 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.274308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.274280 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.274527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.274444 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.274580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.274559 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78gw\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-kube-api-access-m78gw\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.274626 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.274589 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.363154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.363115 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2"] Nov 02 22:11:29.363503 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:11:29.363464 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedf36169_2951_4b4a_a8bc_20b19196adaf.slice/crio-8a7e188d8641123f9d8e173f6e8ba0a02199d3bf47679d4399d8b3cb17e57995 WatchSource:0}: Error finding container 8a7e188d8641123f9d8e173f6e8ba0a02199d3bf47679d4399d8b3cb17e57995: Status 404 returned error can't find the container with id 8a7e188d8641123f9d8e173f6e8ba0a02199d3bf47679d4399d8b3cb17e57995 Nov 02 22:11:29.375908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.375881 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.375923 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.375956 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376062 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78gw\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-kube-api-access-m78gw\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376089 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376153 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376187 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376357 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376414 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376459 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376513 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376669 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376779 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.376862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.376843 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.378321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.378300 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.378894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.378872 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.390444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.390417 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.390761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.390745 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78gw\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-kube-api-access-m78gw\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.564291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.564259 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:29.855844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.855818 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj"] Nov 02 22:11:29.856557 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:11:29.856528 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec22cb4_0c48_427d_af9a_8ebc8607be17.slice/crio-fe6a8dea249eca881e20ee6efa55e238d0299b277fab331c9e177df3301f849e WatchSource:0}: Error finding container fe6a8dea249eca881e20ee6efa55e238d0299b277fab331c9e177df3301f849e: Status 404 returned error can't find the container with id fe6a8dea249eca881e20ee6efa55e238d0299b277fab331c9e177df3301f849e Nov 02 22:11:29.859134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.859101 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:29.859229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.859168 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:29.859229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:29.859216 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:30.106540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:30.106502 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" event={"ID":"edf36169-2951-4b4a-a8bc-20b19196adaf","Type":"ContainerStarted","Data":"a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c"} Nov 02 22:11:30.107049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:30.106556 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" event={"ID":"edf36169-2951-4b4a-a8bc-20b19196adaf","Type":"ContainerStarted","Data":"8a7e188d8641123f9d8e173f6e8ba0a02199d3bf47679d4399d8b3cb17e57995"} Nov 02 22:11:30.108611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:30.108583 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" event={"ID":"5ec22cb4-0c48-427d-af9a-8ebc8607be17","Type":"ContainerStarted","Data":"f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778"} Nov 02 22:11:30.108759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:30.108634 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" event={"ID":"5ec22cb4-0c48-427d-af9a-8ebc8607be17","Type":"ContainerStarted","Data":"fe6a8dea249eca881e20ee6efa55e238d0299b277fab331c9e177df3301f849e"} Nov 02 22:11:30.125983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:30.125934 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podStartSLOduration=2.12591802 podStartE2EDuration="2.12591802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:30.121870967 +0000 UTC m=+818.059984186" watchObservedRunningTime="2026-11-02 22:11:30.12591802 +0000 UTC m=+818.064031239" Nov 02 22:11:30.147346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:30.147247 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podStartSLOduration=1.14723012 podStartE2EDuration="1.14723012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:30.141630494 +0000 UTC m=+818.079743714" watchObservedRunningTime="2026-11-02 22:11:30.14723012 +0000 UTC m=+818.085343339" Nov 02 22:11:30.564891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:30.564864 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:30.566657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:30.566633 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:11:30.566773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:30.566716 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 22:11:31.064831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.064801 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:31.564998 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.564967 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:11:31.565360 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.565020 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 22:11:31.697698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.695998 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww"] Nov 02 22:11:31.724572 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.724539 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww"] Nov 02 22:11:31.724747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.724651 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:31.729118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.728907 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:11:31.729118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.729033 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:11:31.729377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.729359 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:11:31.730838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.730818 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:11:31.828480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.828380 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgpb\" (UniqueName: \"kubernetes.io/projected/329404cd-c051-485d-8b30-1e2f949e8ca9-kube-api-access-kdgpb\") pod \"maas-api-revoke-sub-149a37c2-jftww\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:31.828702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.828658 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149a37c2-jftww\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:31.828870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.828842 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149a37c2-jftww\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:31.930543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.930490 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149a37c2-jftww\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:31.930781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.930628 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149a37c2-jftww\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:31.930781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.930664 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgpb\" (UniqueName: \"kubernetes.io/projected/329404cd-c051-485d-8b30-1e2f949e8ca9-kube-api-access-kdgpb\") pod \"maas-api-revoke-sub-149a37c2-jftww\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:31.931368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.931343 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-149a37c2-jftww\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:31.933055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.933026 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-149a37c2-jftww\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:31.938046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:31.938021 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgpb\" (UniqueName: \"kubernetes.io/projected/329404cd-c051-485d-8b30-1e2f949e8ca9-kube-api-access-kdgpb\") pod \"maas-api-revoke-sub-149a37c2-jftww\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:32.050336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:32.050301 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:32.360318 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:11:32.360284 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329404cd_c051_485d_8b30_1e2f949e8ca9.slice/crio-df6ca29597bf558a2c1cab770a37120d83ef6d07761239b41c90a92746b017f0 WatchSource:0}: Error finding container df6ca29597bf558a2c1cab770a37120d83ef6d07761239b41c90a92746b017f0: Status 404 returned error can't find the container with id df6ca29597bf558a2c1cab770a37120d83ef6d07761239b41c90a92746b017f0 Nov 02 22:11:32.362450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:32.362427 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww"] Nov 02 22:11:32.564534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:32.564503 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:11:32.564703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:32.564551 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 22:11:33.141889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:33.141791 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" event={"ID":"329404cd-c051-485d-8b30-1e2f949e8ca9","Type":"ContainerStarted","Data":"0aafa0595c74481c099829af12ddf4caa487df000c25de072a2aa81932cdbfc7"} Nov 02 22:11:33.142263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:33.141899 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" event={"ID":"329404cd-c051-485d-8b30-1e2f949e8ca9","Type":"ContainerStarted","Data":"df6ca29597bf558a2c1cab770a37120d83ef6d07761239b41c90a92746b017f0"} Nov 02 22:11:33.160467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:33.160417 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" podStartSLOduration=2.16040263 podStartE2EDuration="2.16040263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:33.156804577 +0000 UTC m=+821.094917795" watchObservedRunningTime="2026-11-02 22:11:33.16040263 +0000 UTC m=+821.098515852" Nov 02 22:11:33.253848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:33.253815 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:33.266235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:33.266212 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:33.565217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:33.565185 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:11:33.565399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:33.565241 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 22:11:34.150916 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:34.150884 4309 generic.go:356] "Generic (PLEG): container finished" podID="329404cd-c051-485d-8b30-1e2f949e8ca9" containerID="0aafa0595c74481c099829af12ddf4caa487df000c25de072a2aa81932cdbfc7" exitCode=0 Nov 02 22:11:34.151407 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:34.150945 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" event={"ID":"329404cd-c051-485d-8b30-1e2f949e8ca9","Type":"ContainerDied","Data":"0aafa0595c74481c099829af12ddf4caa487df000c25de072a2aa81932cdbfc7"} Nov 02 22:11:34.162592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:34.162566 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:11:34.565226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:34.565192 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:11:34.565400 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:34.565253 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 22:11:35.424903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.424835 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:35.479570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.479540 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-xks-ca\") pod \"329404cd-c051-485d-8b30-1e2f949e8ca9\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " Nov 02 22:11:35.479789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.479651 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329404cd-c051-485d-8b30-1e2f949e8ca9-kube-api-access-kdgpb\" (UniqueName: \"kubernetes.io/projected/329404cd-c051-485d-8b30-1e2f949e8ca9-kube-api-access-kdgpb\") pod \"329404cd-c051-485d-8b30-1e2f949e8ca9\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " Nov 02 22:11:35.479954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.479929 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-openshift-ca\") pod \"329404cd-c051-485d-8b30-1e2f949e8ca9\" (UID: \"329404cd-c051-485d-8b30-1e2f949e8ca9\") " Nov 02 22:11:35.480337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.480313 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-openshift-ca" pod "329404cd-c051-485d-8b30-1e2f949e8ca9" (UID: "329404cd-c051-485d-8b30-1e2f949e8ca9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:11:35.480616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.480595 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:35.498703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.498610 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-xks-ca" pod "329404cd-c051-485d-8b30-1e2f949e8ca9" (UID: "329404cd-c051-485d-8b30-1e2f949e8ca9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:11:35.498991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.498965 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329404cd-c051-485d-8b30-1e2f949e8ca9-kube-api-access-kdgpb" pod "329404cd-c051-485d-8b30-1e2f949e8ca9" (UID: "329404cd-c051-485d-8b30-1e2f949e8ca9"). InnerVolumeSpecName "kube-api-access-kdgpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:11:35.564829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.564798 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:11:35.564980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.564856 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 22:11:35.588699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.581836 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329404cd-c051-485d-8b30-1e2f949e8ca9-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:35.588699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:35.581868 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdgpb\" (UniqueName: \"kubernetes.io/projected/329404cd-c051-485d-8b30-1e2f949e8ca9-kube-api-access-kdgpb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:11:36.169503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:36.169470 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" Nov 02 22:11:36.169503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:36.169489 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww" event={"ID":"329404cd-c051-485d-8b30-1e2f949e8ca9","Type":"ContainerDied","Data":"df6ca29597bf558a2c1cab770a37120d83ef6d07761239b41c90a92746b017f0"} Nov 02 22:11:36.169773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:36.169581 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6ca29597bf558a2c1cab770a37120d83ef6d07761239b41c90a92746b017f0" Nov 02 22:11:36.565129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:36.565096 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 02 22:11:36.565512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:36.565158 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 02 22:11:37.569447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:37.569416 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:37.570393 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:37.570163 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:37.573846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:37.573825 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:11:39.066699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:39.066044 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:39.155959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:39.155907 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:39.416112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:39.415999 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:11:41.612052 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:11:41.612008 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:11:44.310797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.310765 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv"] Nov 02 22:11:44.311620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.311596 4309 cpu_manager.go:414] "containerMap: removing container" podUID="329404cd-c051-485d-8b30-1e2f949e8ca9" containerName="revoke-keys" Nov 02 22:11:44.311809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.311790 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329404cd-c051-485d-8b30-1e2f949e8ca9" containerName="revoke-keys" Nov 02 22:11:44.312015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.312000 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="329404cd-c051-485d-8b30-1e2f949e8ca9" containerName="revoke-keys" Nov 02 22:11:44.316214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.316192 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.333818 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.333790 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv"] Nov 02 22:11:44.400570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.400534 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trvc\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-kube-api-access-9trvc\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.400803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.400719 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.400895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.400830 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8282cc88-823a-429b-9385-478d41cd1bea-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.400895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.400891 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.401011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.400906 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.401011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.400975 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.401128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.401073 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.401185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.401160 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8282cc88-823a-429b-9385-478d41cd1bea-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.401241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.401210 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.503554 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.503517 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.503778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.503609 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8282cc88-823a-429b-9385-478d41cd1bea-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.503778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.503708 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.503778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.503732 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.503964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.503780 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.503964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.503807 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.503964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.503874 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8282cc88-823a-429b-9385-478d41cd1bea-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.503964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.503927 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.504163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.504009 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trvc\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-kube-api-access-9trvc\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.504739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.504713 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.505442 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.505412 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8282cc88-823a-429b-9385-478d41cd1bea-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.506085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.506055 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.506280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.506254 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.506497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.506471 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.510862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.509745 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8282cc88-823a-429b-9385-478d41cd1bea-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.511112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.511091 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.511637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.511615 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trvc\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-kube-api-access-9trvc\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.515617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.515584 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.643456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.643364 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:44.961698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.960224 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv"] Nov 02 22:11:44.965242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.965211 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:44.965326 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.965277 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:44.965326 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:44.965304 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:11:45.014058 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.014020 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr"] Nov 02 22:11:45.021587 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.021428 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.028725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.028665 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5f4a2f\"/\"distinct-2-w1-5f4a2f-kserve-self-signed-certs\"" Nov 02 22:11:45.030610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.030522 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr"] Nov 02 22:11:45.112647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.112608 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n2t7\" (UniqueName: \"kubernetes.io/projected/07d8db7d-7bfb-4774-b8e7-062e282680ae-kube-api-access-9n2t7\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.112864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.112786 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-home\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.112864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.112851 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-model-cache\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.112979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.112877 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-tmp-dir\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.113032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.112986 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d8db7d-7bfb-4774-b8e7-062e282680ae-tls-certs\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.113032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.113021 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-dshm\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.214005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.213907 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-dshm\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.214238 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.214213 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2t7\" (UniqueName: \"kubernetes.io/projected/07d8db7d-7bfb-4774-b8e7-062e282680ae-kube-api-access-9n2t7\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.214326 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.214302 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-home\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.214406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.214369 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-model-cache\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.214406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.214394 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-tmp-dir\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.214527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.214432 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d8db7d-7bfb-4774-b8e7-062e282680ae-tls-certs\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.214784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.214754 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-home\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.214930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.214901 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-model-cache\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.215049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.214989 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-tmp-dir\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.216592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.216571 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-dshm\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.217225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.217208 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d8db7d-7bfb-4774-b8e7-062e282680ae-tls-certs\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.222330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.222295 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n2t7\" (UniqueName: \"kubernetes.io/projected/07d8db7d-7bfb-4774-b8e7-062e282680ae-kube-api-access-9n2t7\") pod \"distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.352024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.351986 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:45.370769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.370733 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" event={"ID":"8282cc88-823a-429b-9385-478d41cd1bea","Type":"ContainerStarted","Data":"a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2"} Nov 02 22:11:45.370917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.370791 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" event={"ID":"8282cc88-823a-429b-9385-478d41cd1bea","Type":"ContainerStarted","Data":"041f622e9bde244cb64f1b99e3673508cd01b38172279df1f4a06661af379859"} Nov 02 22:11:45.392658 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.392609 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podStartSLOduration=1.392593593 podStartE2EDuration="1.392593593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:45.389347032 +0000 UTC m=+833.327460250" watchObservedRunningTime="2026-11-02 22:11:45.392593593 +0000 UTC m=+833.330706856" Nov 02 22:11:45.643969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.643927 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:45.644278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:45.644252 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr"] Nov 02 22:11:45.648378 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:11:45.648342 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07d8db7d_7bfb_4774_b8e7_062e282680ae.slice/crio-be6bada33e99044272cb3cb328d31ced31b9dc0b3c68cba6638219f5420d8a8a WatchSource:0}: Error finding container be6bada33e99044272cb3cb328d31ced31b9dc0b3c68cba6638219f5420d8a8a: Status 404 returned error can't find the container with id be6bada33e99044272cb3cb328d31ced31b9dc0b3c68cba6638219f5420d8a8a Nov 02 22:11:46.380467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:46.380430 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" event={"ID":"07d8db7d-7bfb-4774-b8e7-062e282680ae","Type":"ContainerStarted","Data":"974433e3f8961267ff849e24d6fd83d4ce9fd3547364968b0e5d50d5253b887a"} Nov 02 22:11:46.380897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:46.380477 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" event={"ID":"07d8db7d-7bfb-4774-b8e7-062e282680ae","Type":"ContainerStarted","Data":"be6bada33e99044272cb3cb328d31ced31b9dc0b3c68cba6638219f5420d8a8a"} Nov 02 22:11:46.420008 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:46.419937 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podStartSLOduration=2.419916891 podStartE2EDuration="2.419916891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:11:46.408104963 +0000 UTC m=+834.346218181" watchObservedRunningTime="2026-11-02 22:11:46.419916891 +0000 UTC m=+834.358030111" Nov 02 22:11:46.645551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:46.645463 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:11:46.645551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:46.645519 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:11:47.352218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:47.352184 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:47.644233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:47.644145 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:11:47.644233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:47.644204 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:11:48.645930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:48.645078 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:11:48.645930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:48.645162 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:11:49.644495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:49.644459 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:11:49.644732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:49.644519 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:11:50.644481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:50.644442 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:11:50.644875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:50.644494 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:11:51.644807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:51.644772 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 22:11:51.645205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:51.644834 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 22:11:52.648860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:52.648815 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:52.649324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:52.649303 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:52.650264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:52.650244 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:11:55.364739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:55.363415 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:55.425037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:55.425004 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:55.528532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:55.528500 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:11:59.566235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.566197 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5"] Nov 02 22:11:59.572452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.572426 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.597860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.597833 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5"] Nov 02 22:11:59.605347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.605294 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp"] Nov 02 22:11:59.612470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.612340 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.614759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.614502 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.615629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.614991 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.615629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.615060 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.615629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.615101 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef3244-ef74-4e12-ae01-22a50f373880-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.615629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.615152 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.615629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.615184 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.615629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.615220 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.615629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.615277 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef3244-ef74-4e12-ae01-22a50f373880-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.615629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.615300 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzhtm\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-kube-api-access-dzhtm\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.643185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.640504 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp"] Nov 02 22:11:59.679385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.674518 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79"] Nov 02 22:11:59.684819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.683914 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.703431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.703405 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79"] Nov 02 22:11:59.716323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716290 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.716434 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716346 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef3244-ef74-4e12-ae01-22a50f373880-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.716434 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716376 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzhtm\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-kube-api-access-dzhtm\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.716510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716435 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.716510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716474 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.716510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716504 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28244\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-kube-api-access-28244\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.716605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716548 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.716605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716576 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.716707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716609 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8qn\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-kube-api-access-7b8qn\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.716707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716636 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.716817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716710 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.716817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716739 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d03196-1dc0-4900-a4a4-a85e289badd4-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.716817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716776 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff4254-f98d-427f-be0a-440c773b870c-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.717055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716827 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.717055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.716881 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.717317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717289 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.717454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717348 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.717454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717390 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.717454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717417 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.717454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717440 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff4254-f98d-427f-be0a-440c773b870c-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.717737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717502 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.717737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717552 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef3244-ef74-4e12-ae01-22a50f373880-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.717737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717597 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.717737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717646 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.717737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717708 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.718045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717757 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.718045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.717795 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.718846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.718823 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.719833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.719807 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef3244-ef74-4e12-ae01-22a50f373880-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.720344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.720318 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.720657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.720636 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.720919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.720898 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.723317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.723295 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.724034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.724011 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef3244-ef74-4e12-ae01-22a50f373880-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.728911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.728658 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzhtm\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-kube-api-access-dzhtm\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.729895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.729874 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.818959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.818882 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.818959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.818930 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.818972 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.819174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.818999 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28244\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-kube-api-access-28244\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819029 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819048 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819388 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819247 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8qn\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-kube-api-access-7b8qn\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.819443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819374 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.819443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819404 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819442 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819670 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819767 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819797 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d03196-1dc0-4900-a4a4-a85e289badd4-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819702 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.819994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819838 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff4254-f98d-427f-be0a-440c773b870c-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.819994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819888 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.819994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819922 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.819994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819937 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.820275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.819996 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.820275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.820045 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.820275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.820071 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.820275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.820095 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff4254-f98d-427f-be0a-440c773b870c-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.820275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.820185 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.820555 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.820346 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.820555 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.820500 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d03196-1dc0-4900-a4a4-a85e289badd4-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.820791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.820763 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.820928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.820903 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.821421 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.821397 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff4254-f98d-427f-be0a-440c773b870c-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.822554 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.822527 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff4254-f98d-427f-be0a-440c773b870c-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.822719 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.822696 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.822775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.822733 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.823489 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.823470 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.834447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.834425 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.834447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.834436 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28244\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-kube-api-access-28244\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.834564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.834512 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8qn\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-kube-api-access-7b8qn\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:11:59.834957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.834936 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:11:59.914113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.914078 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:11:59.965798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:11:59.965540 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:12:00.016694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.014237 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:12:00.145381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.145357 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2"] Nov 02 22:12:00.152252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.152203 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.154828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.154278 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2"] Nov 02 22:12:00.171345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.160655 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c302b6\"/\"distinct-2-w4-c302b6-kserve-self-signed-certs\"" Nov 02 22:12:00.223295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.223263 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5"] Nov 02 22:12:00.224868 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:00.224832 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ef3244_ef74_4e12_ae01_22a50f373880.slice/crio-1c0479ecf3365c6bb8667934403ca3d8c16974214e9d920bc12ee50264a7855d WatchSource:0}: Error finding container 1c0479ecf3365c6bb8667934403ca3d8c16974214e9d920bc12ee50264a7855d: Status 404 returned error can't find the container with id 1c0479ecf3365c6bb8667934403ca3d8c16974214e9d920bc12ee50264a7855d Nov 02 22:12:00.228390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.228350 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:00.228482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.228437 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:00.228550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.228478 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:00.231136 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.231104 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-home\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.231244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.231198 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae16f350-d7fa-44dd-bdff-4493c75156b8-tls-certs\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.231310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.231242 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-dshm\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.231364 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.231321 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-tmp-dir\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.231449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.231434 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt92k\" (UniqueName: \"kubernetes.io/projected/ae16f350-d7fa-44dd-bdff-4493c75156b8-kube-api-access-wt92k\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.231587 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.231571 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-model-cache\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.359890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.343281 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt92k\" (UniqueName: \"kubernetes.io/projected/ae16f350-d7fa-44dd-bdff-4493c75156b8-kube-api-access-wt92k\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.361770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.361716 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-model-cache\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.362158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.362128 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-home\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.362461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.362441 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae16f350-d7fa-44dd-bdff-4493c75156b8-tls-certs\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.362637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.362608 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-dshm\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.362907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.362879 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-tmp-dir\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.363571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.363526 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-tmp-dir\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.364082 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.364049 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-home\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.364904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.364884 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-model-cache\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.367107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.367058 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-dshm\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.367270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.367244 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt92k\" (UniqueName: \"kubernetes.io/projected/ae16f350-d7fa-44dd-bdff-4493c75156b8-kube-api-access-wt92k\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.372286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.372210 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae16f350-d7fa-44dd-bdff-4493c75156b8-tls-certs\") pod \"distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.405738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.402593 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:00.405738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.402658 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:00.405738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.402735 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:00.410580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.410543 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:00.410808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.410786 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:00.410943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.410926 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:00.415125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.412093 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp"] Nov 02 22:12:00.418144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.418119 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79"] Nov 02 22:12:00.516701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.512146 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:00.567116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.566102 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" event={"ID":"e7ff4254-f98d-427f-be0a-440c773b870c","Type":"ContainerStarted","Data":"d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93"} Nov 02 22:12:00.567116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.566166 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" event={"ID":"e7ff4254-f98d-427f-be0a-440c773b870c","Type":"ContainerStarted","Data":"f87a17dae955ebb03bbeb14f35b89a891105d3f479b6d49aa14540a916cb95b3"} Nov 02 22:12:00.603184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.601510 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" event={"ID":"31ef3244-ef74-4e12-ae01-22a50f373880","Type":"ContainerStarted","Data":"e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6"} Nov 02 22:12:00.603184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.601634 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" event={"ID":"31ef3244-ef74-4e12-ae01-22a50f373880","Type":"ContainerStarted","Data":"1c0479ecf3365c6bb8667934403ca3d8c16974214e9d920bc12ee50264a7855d"} Nov 02 22:12:00.642417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.642345 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podStartSLOduration=1.642322939 podStartE2EDuration="1.642322939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:00.619341017 +0000 UTC m=+848.557454255" watchObservedRunningTime="2026-11-02 22:12:00.642322939 +0000 UTC m=+848.580436159" Nov 02 22:12:00.664631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.664595 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" event={"ID":"28d03196-1dc0-4900-a4a4-a85e289badd4","Type":"ContainerStarted","Data":"ec836f011e5b5e48ef6a6b0f6bf6e0304d9d3d07d001f31671cca00c9b964d57"} Nov 02 22:12:00.665111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.665039 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" event={"ID":"28d03196-1dc0-4900-a4a4-a85e289badd4","Type":"ContainerStarted","Data":"782d58080d2c62e21ac8a900dd7599aa4c86bd3c6a7bd805b183f6812e7c316d"} Nov 02 22:12:00.680688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.680625 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podStartSLOduration=1.680611662 podStartE2EDuration="1.680611662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:00.678902585 +0000 UTC m=+848.617015802" watchObservedRunningTime="2026-11-02 22:12:00.680611662 +0000 UTC m=+848.618724880" Nov 02 22:12:00.727890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.727799 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podStartSLOduration=1.727777447 podStartE2EDuration="1.727777447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:00.716219536 +0000 UTC m=+848.654332765" watchObservedRunningTime="2026-11-02 22:12:00.727777447 +0000 UTC m=+848.665890666" Nov 02 22:12:00.914614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.914542 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:12:00.942608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.942580 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2"] Nov 02 22:12:00.967031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:00.967007 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:12:01.016507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.016449 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:12:01.016916 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.016668 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:12:01.016916 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.016740 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 22:12:01.217167 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.217088 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj"] Nov 02 22:12:01.223361 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.223332 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.228944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.228916 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj"] Nov 02 22:12:01.229402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.229377 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:01.229568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.229552 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:12:01.229637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.229601 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:01.230190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.230167 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:01.355070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.355038 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5ht\" (UniqueName: \"kubernetes.io/projected/b4264fd4-a060-4e41-b206-c0ae0eda7eef-kube-api-access-zm5ht\") pod \"maas-api-revoke-sub-5f328a37-7nblj\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.355264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.355236 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f328a37-7nblj\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.355501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.355470 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f328a37-7nblj\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.457372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.457332 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f328a37-7nblj\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.457545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.457523 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f328a37-7nblj\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.457646 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.457625 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5ht\" (UniqueName: \"kubernetes.io/projected/b4264fd4-a060-4e41-b206-c0ae0eda7eef-kube-api-access-zm5ht\") pod \"maas-api-revoke-sub-5f328a37-7nblj\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.458515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.458482 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f328a37-7nblj\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.460030 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.460007 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f328a37-7nblj\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.465259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.465233 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5ht\" (UniqueName: \"kubernetes.io/projected/b4264fd4-a060-4e41-b206-c0ae0eda7eef-kube-api-access-zm5ht\") pod \"maas-api-revoke-sub-5f328a37-7nblj\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.562232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.561946 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:01.805700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.789842 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" event={"ID":"ae16f350-d7fa-44dd-bdff-4493c75156b8","Type":"ContainerStarted","Data":"b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc"} Nov 02 22:12:01.805700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.789897 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" event={"ID":"ae16f350-d7fa-44dd-bdff-4493c75156b8","Type":"ContainerStarted","Data":"8b13f14eb9ab0021a20dc2547d0aa98af182c1c56e9f79f8b5ee4f73f05cf1b1"} Nov 02 22:12:01.929766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.921823 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:12:01.929766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.921895 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:12:01.979712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.973898 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:12:01.979712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:01.973960 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:12:02.017236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:02.016808 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:12:02.017236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:02.016866 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 22:12:02.956146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:02.956109 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:12:02.956759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:02.956727 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:12:03.162660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.043968 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:12:03.162660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.050800 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 22:12:03.162660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.051223 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:12:03.162660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.051263 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:12:03.196863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.196783 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:03.223771 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:12:03.223736 4309 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.024s" Nov 02 22:12:03.223993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.223976 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:03.253712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.245314 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podStartSLOduration=3.245292392 podStartE2EDuration="3.245292392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:01.844014273 +0000 UTC m=+849.782127492" watchObservedRunningTime="2026-11-02 22:12:03.245292392 +0000 UTC m=+851.183405615" Nov 02 22:12:03.278743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.278668 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj"] Nov 02 22:12:03.936760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.936389 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:12:03.936760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.936469 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:12:03.985700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.968748 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:12:03.985700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:03.968832 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:12:04.023902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.019129 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:12:04.023902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.019188 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 22:12:04.861755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.847718 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" event={"ID":"b4264fd4-a060-4e41-b206-c0ae0eda7eef","Type":"ContainerStarted","Data":"fb43f0bb8455b86068e88e8bdde84831502643e158fe2f2ce352e4c96b062125"} Nov 02 22:12:04.862250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.854871 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:04.898805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.898767 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m"] Nov 02 22:12:04.939700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.923226 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:12:04.939700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.923283 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:12:04.966329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.966294 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m"] Nov 02 22:12:04.968820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.966336 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:12:04.969052 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.969025 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:12:04.969202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.966918 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:04.976259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:04.976227 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt"] Nov 02 22:12:05.053143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.052199 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:12:05.053143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.052255 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 22:12:05.083042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.082843 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt"] Nov 02 22:12:05.083364 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.083343 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.235517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.180381 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6474c875-ktr2m\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.242028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.241925 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6474c875-ktr2m\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.247793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.247737 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4393f5-84hkt\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.248336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.248227 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fsx\" (UniqueName: \"kubernetes.io/projected/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-kube-api-access-42fsx\") pod \"maas-api-revoke-sub-6474c875-ktr2m\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.248517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.248466 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4393f5-84hkt\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.248770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.248722 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brxn5\" (UniqueName: \"kubernetes.io/projected/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-kube-api-access-brxn5\") pod \"maas-api-revoke-sub-eb4393f5-84hkt\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.359701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.359607 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brxn5\" (UniqueName: \"kubernetes.io/projected/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-kube-api-access-brxn5\") pod \"maas-api-revoke-sub-eb4393f5-84hkt\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.414299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.371771 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6474c875-ktr2m\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.414299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.371948 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6474c875-ktr2m\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.414299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.372065 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4393f5-84hkt\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.414299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.372151 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fsx\" (UniqueName: \"kubernetes.io/projected/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-kube-api-access-42fsx\") pod \"maas-api-revoke-sub-6474c875-ktr2m\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.414299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.372225 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4393f5-84hkt\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.414299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.374020 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6474c875-ktr2m\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.414299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.377338 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6474c875-ktr2m\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.414299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.378034 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb4393f5-84hkt\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.422802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.422729 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb4393f5-84hkt\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.436328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.431709 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brxn5\" (UniqueName: \"kubernetes.io/projected/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-kube-api-access-brxn5\") pod \"maas-api-revoke-sub-eb4393f5-84hkt\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.440700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.437150 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fsx\" (UniqueName: \"kubernetes.io/projected/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-kube-api-access-42fsx\") pod \"maas-api-revoke-sub-6474c875-ktr2m\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.533181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.528920 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:05.610652 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.610622 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:05.949700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.945803 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:12:05.949700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.945868 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:12:05.971537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.967085 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:12:05.971537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:05.967159 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:12:06.013689 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.011034 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:06.066337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.060285 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:12:06.066337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.060338 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 22:12:06.187730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.170145 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:06.187730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.171151 4309 generic.go:356] "Generic (PLEG): container finished" podID="b4264fd4-a060-4e41-b206-c0ae0eda7eef" containerID="accec6faaf86f93df9755517facd0f8d38d634ceaa18532c310cd237d67b40b2" exitCode=0 Nov 02 22:12:06.187730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.171349 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" event={"ID":"b4264fd4-a060-4e41-b206-c0ae0eda7eef","Type":"ContainerDied","Data":"accec6faaf86f93df9755517facd0f8d38d634ceaa18532c310cd237d67b40b2"} Nov 02 22:12:06.191742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.189242 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:12:06.191742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.190905 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:12:06.477172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.476062 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:06.633761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.532458 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:06.633761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.621163 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:06.676204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.676152 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g"] Nov 02 22:12:06.864850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:06.680036 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:07.036045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.036002 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:12:07.036270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.036235 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:12:07.126701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.052707 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:12:07.126701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.052783 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 22:12:07.126701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.052901 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:12:07.126701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.052938 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:12:07.251940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.229483 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Nov 02 22:12:07.251940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.229586 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:12:07.291452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.257483 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:07.349316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.347218 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:07.349316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.348585 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g"] Nov 02 22:12:07.349316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.348719 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:12:07.349316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.348766 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:12:07.349316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.348803 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:12:07.355470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.355445 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.365213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.365190 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:12:07.387112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.381567 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:12:07.427700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.424628 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c861c0b-vp87g\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.427700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.424716 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c861c0b-vp87g\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.427700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.424987 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbz6\" (UniqueName: \"kubernetes.io/projected/f45f9097-9224-44dc-b3e8-3424cc70e7f8-kube-api-access-gvbz6\") pod \"maas-api-revoke-sub-2c861c0b-vp87g\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.444212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.444177 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m"] Nov 02 22:12:07.449118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.448717 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt"] Nov 02 22:12:07.471593 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.469249 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" event={"ID":"ac2648d5-b7a0-4fa3-a83a-1490ae575afc","Type":"ContainerStarted","Data":"0fe4ba293067d107db31a97bccf6f63b37d7c60af75f9a2262617eca8325d8e1"} Nov 02 22:12:07.482125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.482097 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:12:07.484658 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.484597 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" event={"ID":"2c28dd1d-d2df-47a7-a1b5-8c15117cde76","Type":"ContainerStarted","Data":"ff1fee6fc135488b4e0ed5732f746e47509aa661a8b152595c813c56976cc862"} Nov 02 22:12:07.532508 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.532424 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbz6\" (UniqueName: \"kubernetes.io/projected/f45f9097-9224-44dc-b3e8-3424cc70e7f8-kube-api-access-gvbz6\") pod \"maas-api-revoke-sub-2c861c0b-vp87g\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.533225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.533195 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c861c0b-vp87g\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.533406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.533389 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c861c0b-vp87g\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.534296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.534271 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c861c0b-vp87g\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.551341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.551298 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c861c0b-vp87g\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.571580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.563263 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbz6\" (UniqueName: \"kubernetes.io/projected/f45f9097-9224-44dc-b3e8-3424cc70e7f8-kube-api-access-gvbz6\") pod \"maas-api-revoke-sub-2c861c0b-vp87g\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.782026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.781752 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:07.888240 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.887788 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:07.915415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.914966 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 22:12:07.915415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.915025 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 22:12:07.967721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.966513 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 22:12:07.967721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.966557 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 22:12:07.979690 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.979648 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4264fd4-a060-4e41-b206-c0ae0eda7eef-kube-api-access-zm5ht\" (UniqueName: \"kubernetes.io/projected/b4264fd4-a060-4e41-b206-c0ae0eda7eef-kube-api-access-zm5ht\") pod \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " Nov 02 22:12:07.979878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.979778 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-openshift-ca\") pod \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " Nov 02 22:12:07.979878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.979814 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-xks-ca\") pod \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\" (UID: \"b4264fd4-a060-4e41-b206-c0ae0eda7eef\") " Nov 02 22:12:07.980651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.980599 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-openshift-ca" pod "b4264fd4-a060-4e41-b206-c0ae0eda7eef" (UID: "b4264fd4-a060-4e41-b206-c0ae0eda7eef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:07.983980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.983925 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-xks-ca" pod "b4264fd4-a060-4e41-b206-c0ae0eda7eef" (UID: "b4264fd4-a060-4e41-b206-c0ae0eda7eef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:07.987406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:07.987365 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4264fd4-a060-4e41-b206-c0ae0eda7eef-kube-api-access-zm5ht" pod "b4264fd4-a060-4e41-b206-c0ae0eda7eef" (UID: "b4264fd4-a060-4e41-b206-c0ae0eda7eef"). InnerVolumeSpecName "kube-api-access-zm5ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:08.021010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.019400 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 02 22:12:08.021010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.019442 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 02 22:12:08.082446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.082341 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:08.082446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.082384 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4264fd4-a060-4e41-b206-c0ae0eda7eef-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:08.082446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.082402 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5ht\" (UniqueName: \"kubernetes.io/projected/b4264fd4-a060-4e41-b206-c0ae0eda7eef-kube-api-access-zm5ht\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:08.153458 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:08.153400 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45f9097_9224_44dc_b3e8_3424cc70e7f8.slice/crio-82105c2f269a6c0ecc05c04d1e7796867088ff5f4184154f87bd41b1d811a1e4 WatchSource:0}: Error finding container 82105c2f269a6c0ecc05c04d1e7796867088ff5f4184154f87bd41b1d811a1e4: Status 404 returned error can't find the container with id 82105c2f269a6c0ecc05c04d1e7796867088ff5f4184154f87bd41b1d811a1e4 Nov 02 22:12:08.159485 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.159453 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g"] Nov 02 22:12:08.511919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.511883 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" event={"ID":"b4264fd4-a060-4e41-b206-c0ae0eda7eef","Type":"ContainerDied","Data":"fb43f0bb8455b86068e88e8bdde84831502643e158fe2f2ce352e4c96b062125"} Nov 02 22:12:08.512150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.512129 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb43f0bb8455b86068e88e8bdde84831502643e158fe2f2ce352e4c96b062125" Nov 02 22:12:08.512785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.512765 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj" Nov 02 22:12:08.520534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.519518 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" event={"ID":"ac2648d5-b7a0-4fa3-a83a-1490ae575afc","Type":"ContainerStarted","Data":"f3d510132c319077c736c140fc114bb3bf2d8ee9f322fc326e6c0277fb61ea38"} Nov 02 22:12:08.522051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.522021 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" event={"ID":"2c28dd1d-d2df-47a7-a1b5-8c15117cde76","Type":"ContainerStarted","Data":"e6a8c50fbdce5fa2180b06d6306ab208928ec41c93fcdb3e258a01e3d61be9cd"} Nov 02 22:12:08.525861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.524190 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" event={"ID":"f45f9097-9224-44dc-b3e8-3424cc70e7f8","Type":"ContainerStarted","Data":"82105c2f269a6c0ecc05c04d1e7796867088ff5f4184154f87bd41b1d811a1e4"} Nov 02 22:12:08.558787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.557800 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" podStartSLOduration=4.557781598 podStartE2EDuration="4.557781598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:08.553380845 +0000 UTC m=+856.491494064" watchObservedRunningTime="2026-11-02 22:12:08.557781598 +0000 UTC m=+856.495894817" Nov 02 22:12:08.601578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:08.580921 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" podStartSLOduration=4.580901169 podStartE2EDuration="4.580901169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:08.572264972 +0000 UTC m=+856.510378191" watchObservedRunningTime="2026-11-02 22:12:08.580901169 +0000 UTC m=+856.519014388" Nov 02 22:12:09.021837 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.021753 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:12:09.025894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.025864 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:12:09.030048 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.027852 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:12:09.335787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.335750 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:12:09.338065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.338037 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:12:09.339758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.339736 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:12:09.473830 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.473801 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:12:09.474371 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.474353 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:12:09.475177 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.475152 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:12:09.534714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.534635 4309 generic.go:356] "Generic (PLEG): container finished" podID="ac2648d5-b7a0-4fa3-a83a-1490ae575afc" containerID="f3d510132c319077c736c140fc114bb3bf2d8ee9f322fc326e6c0277fb61ea38" exitCode=0 Nov 02 22:12:09.534882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.534834 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" event={"ID":"ac2648d5-b7a0-4fa3-a83a-1490ae575afc","Type":"ContainerDied","Data":"f3d510132c319077c736c140fc114bb3bf2d8ee9f322fc326e6c0277fb61ea38"} Nov 02 22:12:09.536891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.536869 4309 generic.go:356] "Generic (PLEG): container finished" podID="2c28dd1d-d2df-47a7-a1b5-8c15117cde76" containerID="e6a8c50fbdce5fa2180b06d6306ab208928ec41c93fcdb3e258a01e3d61be9cd" exitCode=0 Nov 02 22:12:09.537123 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.537100 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" event={"ID":"2c28dd1d-d2df-47a7-a1b5-8c15117cde76","Type":"ContainerDied","Data":"e6a8c50fbdce5fa2180b06d6306ab208928ec41c93fcdb3e258a01e3d61be9cd"} Nov 02 22:12:09.541719 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.539439 4309 generic.go:356] "Generic (PLEG): container finished" podID="f45f9097-9224-44dc-b3e8-3424cc70e7f8" containerID="0e4fd11ec9610343659ebfc57249168f02a2e87ecbbf9dbc557a3d7b42dfb5c0" exitCode=0 Nov 02 22:12:09.541719 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:09.539703 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" event={"ID":"f45f9097-9224-44dc-b3e8-3424cc70e7f8","Type":"ContainerDied","Data":"0e4fd11ec9610343659ebfc57249168f02a2e87ecbbf9dbc557a3d7b42dfb5c0"} Nov 02 22:12:10.512611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.512569 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:10.531106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.531036 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:10.562648 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.562615 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:10.797412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.797081 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:10.866544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.866471 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:10.866733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.866600 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-xks-ca\") pod \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " Nov 02 22:12:10.866733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.866669 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-openshift-ca\") pod \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " Nov 02 22:12:10.866874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.866846 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45f9097-9224-44dc-b3e8-3424cc70e7f8-kube-api-access-gvbz6\" (UniqueName: \"kubernetes.io/projected/f45f9097-9224-44dc-b3e8-3424cc70e7f8-kube-api-access-gvbz6\") pod \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\" (UID: \"f45f9097-9224-44dc-b3e8-3424cc70e7f8\") " Nov 02 22:12:10.867198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.867167 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-openshift-ca" pod "f45f9097-9224-44dc-b3e8-3424cc70e7f8" (UID: "f45f9097-9224-44dc-b3e8-3424cc70e7f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:10.867727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.867663 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:10.869343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.869314 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-xks-ca" pod "f45f9097-9224-44dc-b3e8-3424cc70e7f8" (UID: "f45f9097-9224-44dc-b3e8-3424cc70e7f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:10.869467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.869452 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45f9097-9224-44dc-b3e8-3424cc70e7f8-kube-api-access-gvbz6" pod "f45f9097-9224-44dc-b3e8-3424cc70e7f8" (UID: "f45f9097-9224-44dc-b3e8-3424cc70e7f8"). InnerVolumeSpecName "kube-api-access-gvbz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:10.869592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.869574 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:10.969088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.969057 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-kube-api-access-brxn5\" (UniqueName: \"kubernetes.io/projected/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-kube-api-access-brxn5\") pod \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " Nov 02 22:12:10.969243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.969106 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-xks-ca\") pod \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " Nov 02 22:12:10.969243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.969210 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-openshift-ca\") pod \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " Nov 02 22:12:10.969358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.969275 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-xks-ca\") pod \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\" (UID: \"ac2648d5-b7a0-4fa3-a83a-1490ae575afc\") " Nov 02 22:12:10.969420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.969355 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-kube-api-access-42fsx\" (UniqueName: \"kubernetes.io/projected/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-kube-api-access-42fsx\") pod \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " Nov 02 22:12:10.969465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.969421 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-openshift-ca\") pod \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\" (UID: \"2c28dd1d-d2df-47a7-a1b5-8c15117cde76\") " Nov 02 22:12:10.970202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.969651 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-openshift-ca" pod "ac2648d5-b7a0-4fa3-a83a-1490ae575afc" (UID: "ac2648d5-b7a0-4fa3-a83a-1490ae575afc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:10.970393 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.970370 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45f9097-9224-44dc-b3e8-3424cc70e7f8-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:10.970477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.970390 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-openshift-ca" pod "2c28dd1d-d2df-47a7-a1b5-8c15117cde76" (UID: "2c28dd1d-d2df-47a7-a1b5-8c15117cde76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:10.970477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.970396 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvbz6\" (UniqueName: \"kubernetes.io/projected/f45f9097-9224-44dc-b3e8-3424cc70e7f8-kube-api-access-gvbz6\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:10.970477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.970424 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:10.971583 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.971541 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-xks-ca" pod "2c28dd1d-d2df-47a7-a1b5-8c15117cde76" (UID: "2c28dd1d-d2df-47a7-a1b5-8c15117cde76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:10.971856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.971837 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-kube-api-access-brxn5" pod "ac2648d5-b7a0-4fa3-a83a-1490ae575afc" (UID: "ac2648d5-b7a0-4fa3-a83a-1490ae575afc"). InnerVolumeSpecName "kube-api-access-brxn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:10.971923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.971870 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-xks-ca" pod "ac2648d5-b7a0-4fa3-a83a-1490ae575afc" (UID: "ac2648d5-b7a0-4fa3-a83a-1490ae575afc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:10.972089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:10.972069 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-kube-api-access-42fsx" pod "2c28dd1d-d2df-47a7-a1b5-8c15117cde76" (UID: "2c28dd1d-d2df-47a7-a1b5-8c15117cde76"). InnerVolumeSpecName "kube-api-access-42fsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:11.071881 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.071850 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brxn5\" (UniqueName: \"kubernetes.io/projected/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-kube-api-access-brxn5\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:11.071881 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.071882 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:11.072055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.071891 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac2648d5-b7a0-4fa3-a83a-1490ae575afc-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:11.072055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.071900 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42fsx\" (UniqueName: \"kubernetes.io/projected/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-kube-api-access-42fsx\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:11.072055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.071909 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c28dd1d-d2df-47a7-a1b5-8c15117cde76-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:11.557333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.557239 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" Nov 02 22:12:11.557333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.557234 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt" event={"ID":"ac2648d5-b7a0-4fa3-a83a-1490ae575afc","Type":"ContainerDied","Data":"0fe4ba293067d107db31a97bccf6f63b37d7c60af75f9a2262617eca8325d8e1"} Nov 02 22:12:11.557863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.557392 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fe4ba293067d107db31a97bccf6f63b37d7c60af75f9a2262617eca8325d8e1" Nov 02 22:12:11.558817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.558798 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" Nov 02 22:12:11.558923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.558836 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m" event={"ID":"2c28dd1d-d2df-47a7-a1b5-8c15117cde76","Type":"ContainerDied","Data":"ff1fee6fc135488b4e0ed5732f746e47509aa661a8b152595c813c56976cc862"} Nov 02 22:12:11.558923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.558880 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff1fee6fc135488b4e0ed5732f746e47509aa661a8b152595c813c56976cc862" Nov 02 22:12:11.560643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.560612 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" event={"ID":"f45f9097-9224-44dc-b3e8-3424cc70e7f8","Type":"ContainerDied","Data":"82105c2f269a6c0ecc05c04d1e7796867088ff5f4184154f87bd41b1d811a1e4"} Nov 02 22:12:11.560761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.560653 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g" Nov 02 22:12:11.560761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:11.560668 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82105c2f269a6c0ecc05c04d1e7796867088ff5f4184154f87bd41b1d811a1e4" Nov 02 22:12:11.611735 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:12:11.611696 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:12:13.668316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.668284 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg"] Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.669606 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f45f9097-9224-44dc-b3e8-3424cc70e7f8" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.669635 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45f9097-9224-44dc-b3e8-3424cc70e7f8" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.669655 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b4264fd4-a060-4e41-b206-c0ae0eda7eef" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.669664 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4264fd4-a060-4e41-b206-c0ae0eda7eef" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.669731 4309 cpu_manager.go:414] "containerMap: removing container" podUID="ac2648d5-b7a0-4fa3-a83a-1490ae575afc" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.669740 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2648d5-b7a0-4fa3-a83a-1490ae575afc" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.669839 4309 cpu_manager.go:414] "containerMap: removing container" podUID="2c28dd1d-d2df-47a7-a1b5-8c15117cde76" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.669847 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c28dd1d-d2df-47a7-a1b5-8c15117cde76" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.670272 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c28dd1d-d2df-47a7-a1b5-8c15117cde76" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.670292 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45f9097-9224-44dc-b3e8-3424cc70e7f8" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.670308 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4264fd4-a060-4e41-b206-c0ae0eda7eef" containerName="revoke-keys" Nov 02 22:12:13.673610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.670323 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2648d5-b7a0-4fa3-a83a-1490ae575afc" containerName="revoke-keys" Nov 02 22:12:13.684078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.684050 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:13.688080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.688056 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg"] Nov 02 22:12:13.701706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.691277 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:12:13.701706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.691284 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:13.701706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.691322 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:13.701706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.691363 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:13.813248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.813215 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmdh\" (UniqueName: \"kubernetes.io/projected/83d3d023-1380-4758-a6b5-d0c9098d1da0-kube-api-access-xnmdh\") pod \"maas-api-revoke-sub-0c759d5c-8xlgg\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:13.813417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.813262 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c759d5c-8xlgg\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:13.813504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.813478 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c759d5c-8xlgg\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:13.915113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.915076 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmdh\" (UniqueName: \"kubernetes.io/projected/83d3d023-1380-4758-a6b5-d0c9098d1da0-kube-api-access-xnmdh\") pod \"maas-api-revoke-sub-0c759d5c-8xlgg\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:13.915283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.915149 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c759d5c-8xlgg\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:13.915283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.915259 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c759d5c-8xlgg\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:13.915922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.915895 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c759d5c-8xlgg\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:13.917660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.917641 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c759d5c-8xlgg\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:13.925027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:13.924973 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmdh\" (UniqueName: \"kubernetes.io/projected/83d3d023-1380-4758-a6b5-d0c9098d1da0-kube-api-access-xnmdh\") pod \"maas-api-revoke-sub-0c759d5c-8xlgg\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:14.018251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.018215 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:14.336378 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:14.336343 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d3d023_1380_4758_a6b5_d0c9098d1da0.slice/crio-ab18f740e829922e33137c9e9426efe893c64c955088683c7aa781f2a2b43868 WatchSource:0}: Error finding container ab18f740e829922e33137c9e9426efe893c64c955088683c7aa781f2a2b43868: Status 404 returned error can't find the container with id ab18f740e829922e33137c9e9426efe893c64c955088683c7aa781f2a2b43868 Nov 02 22:12:14.337248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.337226 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg"] Nov 02 22:12:14.428126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.428009 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq"] Nov 02 22:12:14.434963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.434932 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.444423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.444391 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq"] Nov 02 22:12:14.460883 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.457127 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx"] Nov 02 22:12:14.464034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.464007 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.473611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.473578 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx"] Nov 02 22:12:14.513728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.510304 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck"] Nov 02 22:12:14.516363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.516133 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.535938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.530237 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.535938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.530317 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.535938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.530359 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.535938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.530386 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lqdk\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-kube-api-access-8lqdk\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.535938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.530485 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.535938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.530570 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c3992-5db9-468b-91c1-0adfa7588628-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.535938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.530652 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.535938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.530773 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c3992-5db9-468b-91c1-0adfa7588628-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.535938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.530922 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.542345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.542223 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck"] Nov 02 22:12:14.605186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.605154 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" event={"ID":"83d3d023-1380-4758-a6b5-d0c9098d1da0","Type":"ContainerStarted","Data":"ab18f740e829922e33137c9e9426efe893c64c955088683c7aa781f2a2b43868"} Nov 02 22:12:14.632990 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.632954 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.633157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633007 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed62b2b1-be33-450c-8833-ed15060827cc-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.633157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633040 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97kc\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-kube-api-access-f97kc\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.633157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633067 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.633341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633163 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c3992-5db9-468b-91c1-0adfa7588628-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.633341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633282 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnjwl\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-kube-api-access-cnjwl\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.633452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633343 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.633452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633354 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.633452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633383 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.633613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633461 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.633613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633522 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.633613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633586 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.633785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633634 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.633785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633655 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.633785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633703 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.633934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633803 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c3992-5db9-468b-91c1-0adfa7588628-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.633934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633861 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9ba007-fba2-4721-90a9-b9512043544b-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.633934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633894 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.633934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633920 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.634135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633944 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lqdk\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-kube-api-access-8lqdk\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.634135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633920 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.634135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.633968 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.634135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634038 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.634135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634073 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.634135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634108 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.634435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634183 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.634435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634218 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9ba007-fba2-4721-90a9-b9512043544b-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.634435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634257 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed62b2b1-be33-450c-8833-ed15060827cc-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.634435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634355 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c3992-5db9-468b-91c1-0adfa7588628-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.634435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634405 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.634435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634414 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.634741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.634462 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.635945 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.635918 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.636626 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.636608 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c3992-5db9-468b-91c1-0adfa7588628-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.646558 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.646533 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.646813 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.646794 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lqdk\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-kube-api-access-8lqdk\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.673413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.673381 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7"] Nov 02 22:12:14.689985 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.689951 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7"] Nov 02 22:12:14.690171 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.690082 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.735950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.735921 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9ba007-fba2-4721-90a9-b9512043544b-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.735950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.735961 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed62b2b1-be33-450c-8833-ed15060827cc-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.736234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736179 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.736329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736288 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.736406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736341 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed62b2b1-be33-450c-8833-ed15060827cc-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.736406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736387 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97kc\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-kube-api-access-f97kc\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.736524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736428 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.736578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736538 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnjwl\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-kube-api-access-cnjwl\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.736651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736588 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.736754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736657 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.736754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736703 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.736754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736740 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.736898 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736754 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-certs\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.736898 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736826 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.736898 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736863 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.737035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.736945 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9ba007-fba2-4721-90a9-b9512043544b-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.737035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.737007 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.737117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.737048 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.737117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.737082 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.737838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.737811 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-data\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.738232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.738200 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.738324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.738296 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-credential-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.738324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.738299 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.738621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.738596 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.738741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.738712 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-socket\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.739178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.739149 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed62b2b1-be33-450c-8833-ed15060827cc-istiod-ca-cert\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.739879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.739855 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.739951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.739915 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-envoy\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.740412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.740374 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9ba007-fba2-4721-90a9-b9512043544b-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.742449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.742423 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9ba007-fba2-4721-90a9-b9512043544b-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.742879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.742857 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.743280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.743259 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed62b2b1-be33-450c-8833-ed15060827cc-istio-podinfo\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.751227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.751200 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.752160 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.752141 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnjwl\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-kube-api-access-cnjwl\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.754869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.754846 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-istio-token\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.756296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.756274 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97kc\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-kube-api-access-f97kc\") pod \"e2e-worker-w5-361a44-openshift-default-86959d977-cwpck\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.756499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.756484 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:14.783646 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.783394 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:14.842750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.842719 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:14.844137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.844107 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.844243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.844151 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhgw\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-kube-api-access-cjhgw\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.844243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.844196 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.844366 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.844276 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.844366 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.844342 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.844366 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.844363 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.844527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.844396 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.844527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.844426 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.844527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.844473 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.951694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.951648 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.951850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.951719 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.951850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.951781 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.951850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.951843 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.952014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.951967 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.952101 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.952076 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.952164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.952122 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhgw\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-kube-api-access-cjhgw\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.952215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.952184 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-data\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.952266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.952213 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.952394 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.952371 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.952460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.952446 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-credential-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.972662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.962011 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-certs\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.972662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.962960 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-socket\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.972662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.963441 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istiod-ca-cert\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.990385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.977260 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-token\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.990385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.986134 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-podinfo\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.990385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.987787 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhgw\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-kube-api-access-cjhgw\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:14.992215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:14.992163 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-envoy\") pod \"e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:15.018224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.017764 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:15.113689 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.113623 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.113994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.113957 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.114168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.114143 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq"] Nov 02 22:12:15.114391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.114366 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.144065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.144036 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx"] Nov 02 22:12:15.158994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.158948 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.159245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.159133 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.159327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.159279 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.206415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.205878 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.206415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.205967 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.206415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.206012 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.212946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.211976 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck"] Nov 02 22:12:15.365224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.365147 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7"] Nov 02 22:12:15.371051 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:15.371021 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e82e0c_d7b3_4881_918d_591dc346c9e2.slice/crio-3b27f59d3341cce860ce8f69f921dba0b14f7259f9d1f371b14d46dd3ca4a07e WatchSource:0}: Error finding container 3b27f59d3341cce860ce8f69f921dba0b14f7259f9d1f371b14d46dd3ca4a07e: Status 404 returned error can't find the container with id 3b27f59d3341cce860ce8f69f921dba0b14f7259f9d1f371b14d46dd3ca4a07e Nov 02 22:12:15.374529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.374494 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.374640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.374575 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.374640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.374621 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:15.642377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.641023 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" event={"ID":"b4e82e0c-d7b3-4881-918d-591dc346c9e2","Type":"ContainerStarted","Data":"ae59824fadb39d0f9e385e87350553b244a2c3c4165461189ebd39e8942e3bb8"} Nov 02 22:12:15.642377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.641102 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" event={"ID":"b4e82e0c-d7b3-4881-918d-591dc346c9e2","Type":"ContainerStarted","Data":"3b27f59d3341cce860ce8f69f921dba0b14f7259f9d1f371b14d46dd3ca4a07e"} Nov 02 22:12:15.647900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.647666 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" event={"ID":"ed62b2b1-be33-450c-8833-ed15060827cc","Type":"ContainerStarted","Data":"2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5"} Nov 02 22:12:15.647900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.647769 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" event={"ID":"ed62b2b1-be33-450c-8833-ed15060827cc","Type":"ContainerStarted","Data":"c56d914daaf2df67268e40f55b8a2405f669e542693893636b68a6072f6b3a32"} Nov 02 22:12:15.649954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.649924 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" event={"ID":"542c3992-5db9-468b-91c1-0adfa7588628","Type":"ContainerStarted","Data":"52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0"} Nov 02 22:12:15.650080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.649977 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" event={"ID":"542c3992-5db9-468b-91c1-0adfa7588628","Type":"ContainerStarted","Data":"18bfd9b65ee6d6d16753935cae9520423e85d32b02bfe485feda552caab60626"} Nov 02 22:12:15.652970 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.652937 4309 generic.go:356] "Generic (PLEG): container finished" podID="83d3d023-1380-4758-a6b5-d0c9098d1da0" containerID="8a86b3a436eca343f1787d15efafb45343fcd6cff6417b1bf0ea9b48637a4be0" exitCode=0 Nov 02 22:12:15.653095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.653018 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" event={"ID":"83d3d023-1380-4758-a6b5-d0c9098d1da0","Type":"ContainerDied","Data":"8a86b3a436eca343f1787d15efafb45343fcd6cff6417b1bf0ea9b48637a4be0"} Nov 02 22:12:15.657054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.657012 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" event={"ID":"4b9ba007-fba2-4721-90a9-b9512043544b","Type":"ContainerStarted","Data":"4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40"} Nov 02 22:12:15.657161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.657084 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" event={"ID":"4b9ba007-fba2-4721-90a9-b9512043544b","Type":"ContainerStarted","Data":"c392baf505cc72c3e72bb71131847f3c62890a438ab6f44b2ee5a3f9160a408b"} Nov 02 22:12:15.669880 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.669793 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podStartSLOduration=1.669774248 podStartE2EDuration="1.669774248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:15.660865788 +0000 UTC m=+863.598979008" watchObservedRunningTime="2026-11-02 22:12:15.669774248 +0000 UTC m=+863.607887467" Nov 02 22:12:15.705481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.705413 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podStartSLOduration=1.705397957 podStartE2EDuration="1.705397957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:15.685910153 +0000 UTC m=+863.624023373" watchObservedRunningTime="2026-11-02 22:12:15.705397957 +0000 UTC m=+863.643511176" Nov 02 22:12:15.736031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.735970 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podStartSLOduration=1.735948906 podStartE2EDuration="1.735948906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:15.732006364 +0000 UTC m=+863.670119582" watchObservedRunningTime="2026-11-02 22:12:15.735948906 +0000 UTC m=+863.674062129" Nov 02 22:12:15.756816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.756784 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:15.779404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.779352 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podStartSLOduration=1.779338562 podStartE2EDuration="1.779338562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:15.757176086 +0000 UTC m=+863.695289304" watchObservedRunningTime="2026-11-02 22:12:15.779338562 +0000 UTC m=+863.717451779" Nov 02 22:12:15.784799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.784739 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:15.844066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:15.844017 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:16.021751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:16.021391 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:16.021751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:16.021520 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:16.021751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:16.021571 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:16.794084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:16.776295 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:16.794084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:16.776369 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:16.832698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:16.816870 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:16.832698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:16.816942 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:16.858723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:16.845934 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:16.858723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:16.845996 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:17.034756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:17.034161 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:17.034756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:17.034229 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:17.214702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:17.214123 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:17.334699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:17.324758 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:17.370699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:17.368810 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:17.395701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:17.389769 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:17.945947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:17.945903 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:17.987701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:17.976136 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:12:18.072786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.059764 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:18.072786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.060072 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:18.105722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.104920 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:18.105722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.104986 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:18.120694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.107591 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:18.120694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.107641 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:18.120694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.107733 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:18.120694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.107757 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:18.141697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.141458 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:18.141697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.141512 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:18.142096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.141733 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:18.240758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.239777 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:18.294666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.292816 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:18.379343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.379293 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:18.403376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.401563 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:18.441568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.440220 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:12:18.441568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.440286 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:12:18.441568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.440322 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:12:18.442604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.442581 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:18.686702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.685831 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:18.686702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.686126 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:18.711701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.693146 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" event={"ID":"83d3d023-1380-4758-a6b5-d0c9098d1da0","Type":"ContainerDied","Data":"ab18f740e829922e33137c9e9426efe893c64c955088683c7aa781f2a2b43868"} Nov 02 22:12:18.711701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.693200 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab18f740e829922e33137c9e9426efe893c64c955088683c7aa781f2a2b43868" Nov 02 22:12:18.711701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.705132 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-openshift-ca\") pod \"83d3d023-1380-4758-a6b5-d0c9098d1da0\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " Nov 02 22:12:18.711701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.705235 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-xks-ca\") pod \"83d3d023-1380-4758-a6b5-d0c9098d1da0\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " Nov 02 22:12:18.711701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.705567 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d3d023-1380-4758-a6b5-d0c9098d1da0-kube-api-access-xnmdh\" (UniqueName: \"kubernetes.io/projected/83d3d023-1380-4758-a6b5-d0c9098d1da0-kube-api-access-xnmdh\") pod \"83d3d023-1380-4758-a6b5-d0c9098d1da0\" (UID: \"83d3d023-1380-4758-a6b5-d0c9098d1da0\") " Nov 02 22:12:18.736699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.736168 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-openshift-ca" pod "83d3d023-1380-4758-a6b5-d0c9098d1da0" (UID: "83d3d023-1380-4758-a6b5-d0c9098d1da0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:18.829700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.812880 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:18.872848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.872811 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:18.967255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.967155 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:18.967842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.967761 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:12:18.968000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.873775 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:18.968183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.968161 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:18.971814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.971789 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:12:19.087694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.978373 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:19.125706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:18.978410 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 02 22:12:19.125706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.023058 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:19.125706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.124843 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:19.125706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.058782 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-xks-ca" pod "83d3d023-1380-4758-a6b5-d0c9098d1da0" (UID: "83d3d023-1380-4758-a6b5-d0c9098d1da0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:19.125706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.102828 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d3d023-1380-4758-a6b5-d0c9098d1da0-kube-api-access-xnmdh" pod "83d3d023-1380-4758-a6b5-d0c9098d1da0" (UID: "83d3d023-1380-4758-a6b5-d0c9098d1da0"). InnerVolumeSpecName "kube-api-access-xnmdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:19.125706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.125260 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:19.126156 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.072420 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:19.132698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.126489 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:12:19.160704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.153127 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:19.221090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.194365 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:12:19.269371 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.242721 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:19.290758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.272109 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmdh\" (UniqueName: \"kubernetes.io/projected/83d3d023-1380-4758-a6b5-d0c9098d1da0-kube-api-access-xnmdh\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:19.433220 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.433191 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d3d023-1380-4758-a6b5-d0c9098d1da0-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:19.433429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.305974 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:19.453700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.407754 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:19.454248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.407791 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:19.454639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.304769 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:19.455114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.455091 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:12:19.456541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.456507 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:19.553756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.549920 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:19.602466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.602422 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:19.705699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.705447 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:19.706694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.706496 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:12:19.797718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.786134 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:19.797718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.786193 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:19.808113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.808084 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:19.810543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.810507 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:19.939891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.886799 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg" Nov 02 22:12:19.940289 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.899280 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:19.940465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.940440 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:19.984699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:19.982818 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:20.075054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.074952 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:20.075620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.075589 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:20.075819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.075796 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:20.130694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.130251 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:20.130694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.130575 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:20.205476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.205418 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:20.248314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.247251 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:20.425336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.421963 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:20.454987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.454936 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:20.472738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.460767 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:12:20.472738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.462425 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:20.472738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.466481 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:20.472738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.467177 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:12:20.704980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.704873 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:20.761701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.758257 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:12:20.899666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.836448 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:20.899666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.836505 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:20.899666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.879801 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:20.899666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.879869 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:21.006788 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:20.986199 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:21.101886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.076615 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:21.101886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.076692 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:21.182704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.167034 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:12:21.220984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.206502 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:21.220984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.206580 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:21.254996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.254958 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:12:21.288813 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.288713 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:12:21.316843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.316795 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:12:21.375710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.345459 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:21.386702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.370526 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:21.421700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.370665 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:21.524871 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.479215 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:21.524871 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.479630 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:12:21.528694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.528594 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Nov 02 22:12:21.561997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.529047 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:21.786114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.772240 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:21.997933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.881871 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:21.997933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.987795 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:21.997933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.988556 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:21.997933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.988611 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:21.997933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.988946 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:21.997933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:21.988994 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:22.018668 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.018633 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:22.044100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.044060 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:22.272277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.272177 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:12:22.518208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.335998 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:22.518725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.398825 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:12:22.518896 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.398868 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:12:22.519068 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.458863 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:22.519190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.481092 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:22.519293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.492196 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:22.519519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.492757 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:22.535761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.533874 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:22.783062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.783024 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:22.789752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.783274 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:22.801696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.794757 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:22.813799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.813750 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:22.860039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.860001 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:22.889818 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.889772 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:22.919780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.919733 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:22.987768 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:22.975082 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:23.015819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.015773 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:12:23.055760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.052469 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:23.075592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.074744 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:23.625700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.625636 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:23.629702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.627024 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:23.629702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.627330 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:12:23.637701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.633127 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:23.637701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.633392 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:12:23.637701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.633642 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:23.637701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.633888 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:12:23.637701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.634140 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:23.637701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.634348 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:12:23.637701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.634632 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:12:23.761700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.760937 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:23.761700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.761000 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:23.818701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.806177 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:23.818701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.806238 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:23.881763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.874235 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:23.895816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.895766 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:23.941124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.941085 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:23.941391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.941363 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:23.997146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:23.996784 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:24.019937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.019891 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:24.033071 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.020537 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:12:24.071219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.022361 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:24.071814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.071781 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:24.103688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.103633 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:24.104403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.103898 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:24.256045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.254898 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:24.281876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.281523 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:24.322070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.322024 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:24.640947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.640906 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:24.651695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.641807 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:12:24.893098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.649754 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:12:24.915908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.652313 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:12:24.917522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.652344 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:12:24.921870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.652364 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:24.927001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.652382 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:12:24.931055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.652404 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:24.939765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.652428 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:24.943704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.759116 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:24.943967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.943916 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:24.973952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.973907 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:24.974265 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.974214 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:24.976279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.974923 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:24.976279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.975965 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:24.976279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:24.974977 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb"] Nov 02 22:12:25.185864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.167309 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:25.633126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.210844 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:25.633126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.339194 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:25.633126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.624250 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:25.633126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.471968 4309 cpu_manager.go:414] "containerMap: removing container" podUID="83d3d023-1380-4758-a6b5-d0c9098d1da0" containerName="revoke-keys" Nov 02 22:12:25.633126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.624338 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d3d023-1380-4758-a6b5-d0c9098d1da0" containerName="revoke-keys" Nov 02 22:12:25.633126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.624993 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d3d023-1380-4758-a6b5-d0c9098d1da0" containerName="revoke-keys" Nov 02 22:12:25.657707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.656822 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:25.665702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.473052 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:25.665702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.659040 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:25.690931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.690899 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb"] Nov 02 22:12:25.858704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.774932 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:25.862704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.815609 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:25.862704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.859062 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:25.913757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.892350 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbz6s\" (UniqueName: \"kubernetes.io/projected/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-kube-api-access-jbz6s\") pod \"maas-api-revoke-sub-2fd54212-2qfjb\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.096856 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fd54212-2qfjb\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.097165 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fd54212-2qfjb\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.915610 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.914027 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.097498 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.075901 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.914487 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.915728 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:25.915793 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.107551 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.107612 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.108289 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:26.108782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.108339 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:26.157727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.144069 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:26.157727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.144033 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:26.157727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.144922 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:26.157727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.145228 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:26.157727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.145475 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:26.158355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.158324 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:26.166869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.166779 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:26.197718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.158518 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:12:26.240695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.218753 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fd54212-2qfjb\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.382701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.267650 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:12:26.382701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.374604 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fd54212-2qfjb\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.382701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.377553 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fd54212-2qfjb\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.395631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.386881 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbz6s\" (UniqueName: \"kubernetes.io/projected/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-kube-api-access-jbz6s\") pod \"maas-api-revoke-sub-2fd54212-2qfjb\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.423762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.421151 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:26.445630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.445589 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbz6s\" (UniqueName: \"kubernetes.io/projected/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-kube-api-access-jbz6s\") pod \"maas-api-revoke-sub-2fd54212-2qfjb\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.590699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.565876 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fd54212-2qfjb\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.629829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.629784 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:26.635992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.635949 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:26.671026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.670978 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:26.695135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.695040 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:26.840737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.840136 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" podUID="3768a351-a0c8-4472-8769-d39331d91f5a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:26.854099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.845994 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:26.854099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.846050 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:26.854099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.846812 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:26.854099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.846852 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:26.854099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.847036 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:26.854099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:26.847065 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:27.020380 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.020280 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:27.020634 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.020591 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:27.126696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.117981 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:27.126696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.118306 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:27.148707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.146145 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:27.148707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.146489 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:12:27.148707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.146763 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:12:27.148707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.146991 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:12:27.149717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.149689 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:27.200700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.189525 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:27.396696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.393437 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:27.816953 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.797129 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:27.816953 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.799747 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:12:27.884920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.884879 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:27.891836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.891792 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:27.946179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.903258 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:28.185699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.161056 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:28.214802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.921321 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:28.214802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:27.925518 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:28.214802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.195012 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:28.214802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.031212 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:28.214802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.195062 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:28.214802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.146757 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:28.241703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.216752 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:28.282911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.218486 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:28.282911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.275406 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:28.282911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.275700 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:12:28.282911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.270588 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:12:28.305701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.234234 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:28.487758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.484578 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:28.631697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.612767 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb"] Nov 02 22:12:28.759166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.759072 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:28.759414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.759368 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:28.797700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.797386 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:28.797700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.797448 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:28.797700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.797523 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:28.822649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.822602 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:28.855706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.851790 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:28.855706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.851851 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:28.910645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:28.910594 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 02 22:12:29.024345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.024223 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:29.024639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.024586 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:29.159788 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.159744 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:29.194874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.194838 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:29.195702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.195653 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:29.253582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.253527 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" event={"ID":"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f","Type":"ContainerStarted","Data":"4bb249ce81af5c1b579cb7ff1dc88c13eaec7a14ad62fe608d011bdf7713b65a"} Nov 02 22:12:29.277248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.277144 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:29.278145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.277457 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:29.279298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.277505 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:29.280031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.277523 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:29.285129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.277550 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:12:29.304721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.284713 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:29.499770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.497779 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:29.793670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.793636 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:29.817753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.793939 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:29.842698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.796252 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:29.842698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.835863 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:29.842698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.806757 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:29.842698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.818134 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:29.842698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.838065 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:29.881708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.873374 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:29.881708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.879818 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:29.881708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.879864 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:29.881708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.880994 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:12:29.915031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.914984 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:29.920724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:29.916093 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:12:30.064710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.054325 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:30.166857 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.066180 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:12:30.167178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.143663 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:30.279239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.278415 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:12:30.339545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.339037 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:12:30.339545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.339160 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:12:30.339545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.339199 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:12:30.407703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.167088 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:12:30.534613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.534519 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:30.545858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.545831 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:12:30.643416 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.643384 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:12:30.643751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.643731 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:12:30.643947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.643931 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:12:30.644126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.644111 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:12:30.665336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.665296 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" event={"ID":"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f","Type":"ContainerStarted","Data":"ca21569b1aa0ec532bb48b050e81f601c3f1b1b929b038bb7d71f2d877f69f02"} Nov 02 22:12:30.797781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.797692 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 22:12:30.798047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.798016 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 22:12:30.815589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.815554 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 22:12:30.815888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.815858 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 22:12:30.844250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.843598 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 22:12:30.844250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:30.843655 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 22:12:31.447178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:31.447131 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:31.540869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:31.540838 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:12:31.912626 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:31.912592 4309 generic.go:356] "Generic (PLEG): container finished" podID="a6b0c9a5-c63e-4fb9-9272-3a210a448c9f" containerID="ca21569b1aa0ec532bb48b050e81f601c3f1b1b929b038bb7d71f2d877f69f02" exitCode=0 Nov 02 22:12:31.913518 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:31.913489 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" event={"ID":"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f","Type":"ContainerDied","Data":"ca21569b1aa0ec532bb48b050e81f601c3f1b1b929b038bb7d71f2d877f69f02"} Nov 02 22:12:31.972526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:31.972493 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:31.979361 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:31.979286 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:12:32.186535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.184797 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:32.201453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.201419 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:12:32.240153 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.240118 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:32.242928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.242901 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:12:32.351918 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.351886 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:32.356662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.356626 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz"] Nov 02 22:12:32.358913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.358886 4309 cpu_manager.go:414] "containerMap: removing container" podUID="a6b0c9a5-c63e-4fb9-9272-3a210a448c9f" containerName="revoke-keys" Nov 02 22:12:32.359410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.359396 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b0c9a5-c63e-4fb9-9272-3a210a448c9f" containerName="revoke-keys" Nov 02 22:12:32.359729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.359712 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b0c9a5-c63e-4fb9-9272-3a210a448c9f" containerName="revoke-keys" Nov 02 22:12:32.364174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.364158 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.385477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.385450 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz"] Nov 02 22:12:32.435136 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.435044 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-openshift-ca\") pod \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " Nov 02 22:12:32.435419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.435399 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-kube-api-access-jbz6s\" (UniqueName: \"kubernetes.io/projected/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-kube-api-access-jbz6s\") pod \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " Nov 02 22:12:32.435849 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.435822 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-openshift-ca" pod "a6b0c9a5-c63e-4fb9-9272-3a210a448c9f" (UID: "a6b0c9a5-c63e-4fb9-9272-3a210a448c9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:32.436005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.435837 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-xks-ca\") pod \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\" (UID: \"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f\") " Nov 02 22:12:32.436570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.436549 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-067f2141-m5jqz\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.437070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.437047 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-kube-api-access-rdn65\") pod \"maas-api-revoke-sub-067f2141-m5jqz\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.437266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.437248 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-067f2141-m5jqz\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.437492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.437478 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:32.439205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.439175 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-xks-ca" pod "a6b0c9a5-c63e-4fb9-9272-3a210a448c9f" (UID: "a6b0c9a5-c63e-4fb9-9272-3a210a448c9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:32.439445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.439425 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-kube-api-access-jbz6s" pod "a6b0c9a5-c63e-4fb9-9272-3a210a448c9f" (UID: "a6b0c9a5-c63e-4fb9-9272-3a210a448c9f"). InnerVolumeSpecName "kube-api-access-jbz6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:32.538871 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.538834 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-kube-api-access-rdn65\") pod \"maas-api-revoke-sub-067f2141-m5jqz\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.539330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.538886 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-067f2141-m5jqz\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.539330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.538970 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-067f2141-m5jqz\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.539330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.539127 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:32.539330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.539144 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbz6s\" (UniqueName: \"kubernetes.io/projected/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f-kube-api-access-jbz6s\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:32.539868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.539837 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-067f2141-m5jqz\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.542100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.542077 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-067f2141-m5jqz\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.546400 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.546377 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-kube-api-access-rdn65\") pod \"maas-api-revoke-sub-067f2141-m5jqz\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.711456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.711358 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:32.941588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.941556 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" Nov 02 22:12:32.945893 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.941895 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb" event={"ID":"a6b0c9a5-c63e-4fb9-9272-3a210a448c9f","Type":"ContainerDied","Data":"4bb249ce81af5c1b579cb7ff1dc88c13eaec7a14ad62fe608d011bdf7713b65a"} Nov 02 22:12:32.946023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:32.945961 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bb249ce81af5c1b579cb7ff1dc88c13eaec7a14ad62fe608d011bdf7713b65a" Nov 02 22:12:33.018801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.018654 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz"] Nov 02 22:12:33.032477 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:33.031501 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2094c0_6717_4ecf_9b13_c3f6cb176a10.slice/crio-f29847ce7b04fcde9e7f26dfd4dad530916c6b07c148627199ae2424d3323fd5 WatchSource:0}: Error finding container f29847ce7b04fcde9e7f26dfd4dad530916c6b07c148627199ae2424d3323fd5: Status 404 returned error can't find the container with id f29847ce7b04fcde9e7f26dfd4dad530916c6b07c148627199ae2424d3323fd5 Nov 02 22:12:33.829191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.829160 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd"] Nov 02 22:12:33.837720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.837657 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:33.840876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.840847 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd"] Nov 02 22:12:33.868023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.867994 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e943a27e-af4d-4644-a087-392e890a0120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407489b3-llrfd\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:33.868175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.868150 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e943a27e-af4d-4644-a087-392e890a0120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407489b3-llrfd\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:33.868329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.868315 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x84b6\" (UniqueName: \"kubernetes.io/projected/e943a27e-af4d-4644-a087-392e890a0120-kube-api-access-x84b6\") pod \"maas-api-revoke-sub-407489b3-llrfd\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:33.950544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.950511 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" event={"ID":"4e2094c0-6717-4ecf-9b13-c3f6cb176a10","Type":"ContainerStarted","Data":"f7b9d935f86072ead646588cb60506b6de10af0f59d9e4f734b3a9783c479a51"} Nov 02 22:12:33.950760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.950565 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" event={"ID":"4e2094c0-6717-4ecf-9b13-c3f6cb176a10","Type":"ContainerStarted","Data":"f29847ce7b04fcde9e7f26dfd4dad530916c6b07c148627199ae2424d3323fd5"} Nov 02 22:12:33.969430 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.969378 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" podStartSLOduration=1.969364541 podStartE2EDuration="1.969364541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:33.964973096 +0000 UTC m=+881.903086314" watchObservedRunningTime="2026-11-02 22:12:33.969364541 +0000 UTC m=+881.907477758" Nov 02 22:12:33.969600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.969492 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e943a27e-af4d-4644-a087-392e890a0120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407489b3-llrfd\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:33.969784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.969621 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84b6\" (UniqueName: \"kubernetes.io/projected/e943a27e-af4d-4644-a087-392e890a0120-kube-api-access-x84b6\") pod \"maas-api-revoke-sub-407489b3-llrfd\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:33.969947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.969917 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e943a27e-af4d-4644-a087-392e890a0120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407489b3-llrfd\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:33.970324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.970304 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e943a27e-af4d-4644-a087-392e890a0120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407489b3-llrfd\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:33.972492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.972461 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e943a27e-af4d-4644-a087-392e890a0120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407489b3-llrfd\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:33.976908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:33.976883 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84b6\" (UniqueName: \"kubernetes.io/projected/e943a27e-af4d-4644-a087-392e890a0120-kube-api-access-x84b6\") pod \"maas-api-revoke-sub-407489b3-llrfd\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:34.158324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:34.158293 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:34.453920 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:34.453878 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode943a27e_af4d_4644_a087_392e890a0120.slice/crio-e023112db84b099f77a072950c54d06798c7fcc3ac5190196cbbdda665cf86d6 WatchSource:0}: Error finding container e023112db84b099f77a072950c54d06798c7fcc3ac5190196cbbdda665cf86d6: Status 404 returned error can't find the container with id e023112db84b099f77a072950c54d06798c7fcc3ac5190196cbbdda665cf86d6 Nov 02 22:12:34.454373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:34.454353 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd"] Nov 02 22:12:34.960649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:34.960617 4309 generic.go:356] "Generic (PLEG): container finished" podID="4e2094c0-6717-4ecf-9b13-c3f6cb176a10" containerID="f7b9d935f86072ead646588cb60506b6de10af0f59d9e4f734b3a9783c479a51" exitCode=0 Nov 02 22:12:34.961106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:34.960708 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" event={"ID":"4e2094c0-6717-4ecf-9b13-c3f6cb176a10","Type":"ContainerDied","Data":"f7b9d935f86072ead646588cb60506b6de10af0f59d9e4f734b3a9783c479a51"} Nov 02 22:12:34.962516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:34.962493 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" event={"ID":"e943a27e-af4d-4644-a087-392e890a0120","Type":"ContainerStarted","Data":"0cf2cda99907725215d91ba8374c7f72e74357d014fb63cb0e550c777d99ae86"} Nov 02 22:12:34.962600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:34.962538 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" event={"ID":"e943a27e-af4d-4644-a087-392e890a0120","Type":"ContainerStarted","Data":"e023112db84b099f77a072950c54d06798c7fcc3ac5190196cbbdda665cf86d6"} Nov 02 22:12:35.003209 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:35.003124 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" podStartSLOduration=2.003103848 podStartE2EDuration="2.003103848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:34.993811765 +0000 UTC m=+882.931925039" watchObservedRunningTime="2026-11-02 22:12:35.003103848 +0000 UTC m=+882.941217066" Nov 02 22:12:35.974213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:35.974179 4309 generic.go:356] "Generic (PLEG): container finished" podID="e943a27e-af4d-4644-a087-392e890a0120" containerID="0cf2cda99907725215d91ba8374c7f72e74357d014fb63cb0e550c777d99ae86" exitCode=0 Nov 02 22:12:35.974613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:35.974265 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" event={"ID":"e943a27e-af4d-4644-a087-392e890a0120","Type":"ContainerDied","Data":"0cf2cda99907725215d91ba8374c7f72e74357d014fb63cb0e550c777d99ae86"} Nov 02 22:12:36.151080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.151056 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:36.212118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.212091 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-openshift-ca\") pod \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " Nov 02 22:12:36.212338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.212317 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-kube-api-access-rdn65\") pod \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " Nov 02 22:12:36.212437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.212415 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-xks-ca\") pod \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\" (UID: \"4e2094c0-6717-4ecf-9b13-c3f6cb176a10\") " Nov 02 22:12:36.212586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.212567 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-openshift-ca" pod "4e2094c0-6717-4ecf-9b13-c3f6cb176a10" (UID: "4e2094c0-6717-4ecf-9b13-c3f6cb176a10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:36.215309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.215229 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.221964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.221119 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-xks-ca" pod "4e2094c0-6717-4ecf-9b13-c3f6cb176a10" (UID: "4e2094c0-6717-4ecf-9b13-c3f6cb176a10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:36.223150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.223119 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-kube-api-access-rdn65" pod "4e2094c0-6717-4ecf-9b13-c3f6cb176a10" (UID: "4e2094c0-6717-4ecf-9b13-c3f6cb176a10"). InnerVolumeSpecName "kube-api-access-rdn65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:36.316528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.316485 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdn65\" (UniqueName: \"kubernetes.io/projected/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-kube-api-access-rdn65\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.316528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.316516 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2094c0-6717-4ecf-9b13-c3f6cb176a10-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:36.995760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.995716 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" event={"ID":"4e2094c0-6717-4ecf-9b13-c3f6cb176a10","Type":"ContainerDied","Data":"f29847ce7b04fcde9e7f26dfd4dad530916c6b07c148627199ae2424d3323fd5"} Nov 02 22:12:36.996234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.995817 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f29847ce7b04fcde9e7f26dfd4dad530916c6b07c148627199ae2424d3323fd5" Nov 02 22:12:36.996791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:36.996654 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz" Nov 02 22:12:37.175109 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.175086 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:37.229526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.229491 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e943a27e-af4d-4644-a087-392e890a0120-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e943a27e-af4d-4644-a087-392e890a0120-maas-api-xks-ca\") pod \"e943a27e-af4d-4644-a087-392e890a0120\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " Nov 02 22:12:37.229801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.229779 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e943a27e-af4d-4644-a087-392e890a0120-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e943a27e-af4d-4644-a087-392e890a0120-maas-api-openshift-ca\") pod \"e943a27e-af4d-4644-a087-392e890a0120\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " Nov 02 22:12:37.229907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.229829 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e943a27e-af4d-4644-a087-392e890a0120-kube-api-access-x84b6\" (UniqueName: \"kubernetes.io/projected/e943a27e-af4d-4644-a087-392e890a0120-kube-api-access-x84b6\") pod \"e943a27e-af4d-4644-a087-392e890a0120\" (UID: \"e943a27e-af4d-4644-a087-392e890a0120\") " Nov 02 22:12:37.230162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.230137 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e943a27e-af4d-4644-a087-392e890a0120-maas-api-openshift-ca" pod "e943a27e-af4d-4644-a087-392e890a0120" (UID: "e943a27e-af4d-4644-a087-392e890a0120"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:37.230867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.230846 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e943a27e-af4d-4644-a087-392e890a0120-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.232415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.232391 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e943a27e-af4d-4644-a087-392e890a0120-maas-api-xks-ca" pod "e943a27e-af4d-4644-a087-392e890a0120" (UID: "e943a27e-af4d-4644-a087-392e890a0120"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:37.232749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.232715 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e943a27e-af4d-4644-a087-392e890a0120-kube-api-access-x84b6" pod "e943a27e-af4d-4644-a087-392e890a0120" (UID: "e943a27e-af4d-4644-a087-392e890a0120"). InnerVolumeSpecName "kube-api-access-x84b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:37.332325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.332289 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e943a27e-af4d-4644-a087-392e890a0120-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:37.332325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:37.332322 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x84b6\" (UniqueName: \"kubernetes.io/projected/e943a27e-af4d-4644-a087-392e890a0120-kube-api-access-x84b6\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:38.015696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:38.013936 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" event={"ID":"e943a27e-af4d-4644-a087-392e890a0120","Type":"ContainerDied","Data":"e023112db84b099f77a072950c54d06798c7fcc3ac5190196cbbdda665cf86d6"} Nov 02 22:12:38.015696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:38.014005 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e023112db84b099f77a072950c54d06798c7fcc3ac5190196cbbdda665cf86d6" Nov 02 22:12:38.015696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:38.014075 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd" Nov 02 22:12:39.502702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:39.498764 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:39.533701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:39.528973 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:39.633698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:39.632766 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:39.930700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:39.929773 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.005705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.004877 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:40.094699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.086813 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.205758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.205042 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.228701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.217204 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.232698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.230706 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.289836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.287796 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.433808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.433766 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.478955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.478853 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:12:40.528636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.528594 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.530114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.530088 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:12:40.531396 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.531337 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.531847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.531823 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:12:40.533047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.533026 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:12:40.662879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.662833 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:12:40.709697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.684399 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:12:40.942883 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.942839 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:40.944071 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:40.944043 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:12:41.231946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.205175 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:41.232306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.232268 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:12:41.237815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.237783 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:41.301102 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.301051 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:12:41.324180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.304555 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:12:41.324180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.304619 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:12:41.324180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.304661 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:12:41.324180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.304715 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:12:41.512667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.505163 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:41.512667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.506571 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:12:41.529653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.529616 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:41.531248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.531219 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:41.547700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.531874 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:41.664935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.559803 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:41.675700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.671611 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:12:41.692704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.574271 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:12:41.695747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.616547 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" podUID="774397e4-c8ee-4734-8f5a-7befd9b4d9dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:41.696443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.616629 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" podUID="96e4208b-1bbe-4a26-a5c0-77783d8239b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:41.696879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.696860 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:41.697870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.697853 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:12:41.980942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:41.980900 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:42.354242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.354198 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:42.394698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.394432 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:12:42.394879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.394834 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:12:42.469739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.469703 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:42.508060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.508019 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:42.542393 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.542343 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:42.666176 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.666079 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:42.666791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.666759 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:42.701002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.700962 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:42.701636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.701604 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:42.794439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.794390 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Nov 02 22:12:42.985897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:42.985786 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.156051 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:12:43.155991 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:12:43.367696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.357591 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.405702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.394924 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.405702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.395083 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.470126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.470079 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.521618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.519076 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.557093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.557048 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.791771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.774650 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.880701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.814211 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:12:43.931339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.848951 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.932101 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.902862 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:43.945738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.932791 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:12:43.984849 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.984805 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:12:43.985172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:43.985145 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:44.012486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.012437 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:44.279701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.277932 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 02 22:12:44.279701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.278077 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:44.362978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.355087 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:44.408837 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.408786 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:44.410310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.410276 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:12:44.484833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.484782 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:44.577745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.532126 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:44.846841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.565714 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:44.852499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.578781 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:44.897831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.887789 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:44.963211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.963168 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:12:45.224752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:44.995817 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:12:45.225506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:45.060264 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:45.226311 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:45.081386 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:45.262696 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:12:45.260235 4309 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.017s" Nov 02 22:12:45.549113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:45.549006 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:45.564850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:45.557712 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:45.674520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:45.674378 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:12:45.730444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:45.708071 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:12:45.773441 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:45.773410 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:12:45.894692 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:45.890146 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:45.894692 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:45.890493 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:46.089721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.085555 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:12:46.175354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.175265 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:12:46.230980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.230938 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:46.243701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.238481 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:46.262828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.262798 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:12:46.263012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.262843 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh"] Nov 02 22:12:46.264446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.264421 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e943a27e-af4d-4644-a087-392e890a0120" containerName="revoke-keys" Nov 02 22:12:46.264446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.264448 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e943a27e-af4d-4644-a087-392e890a0120" containerName="revoke-keys" Nov 02 22:12:46.264618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.264493 4309 cpu_manager.go:414] "containerMap: removing container" podUID="4e2094c0-6717-4ecf-9b13-c3f6cb176a10" containerName="revoke-keys" Nov 02 22:12:46.264618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.264502 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2094c0-6717-4ecf-9b13-c3f6cb176a10" containerName="revoke-keys" Nov 02 22:12:46.264732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.264711 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2094c0-6717-4ecf-9b13-c3f6cb176a10" containerName="revoke-keys" Nov 02 22:12:46.264780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.264735 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e943a27e-af4d-4644-a087-392e890a0120" containerName="revoke-keys" Nov 02 22:12:46.271401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.271369 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:46.271521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.271405 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh"] Nov 02 22:12:46.271521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.271451 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:12:46.271521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.271483 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx"] Nov 02 22:12:46.277521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.275516 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.278381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.278361 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:12:46.279170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.279150 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:12:46.279313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.278905 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.279427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.279301 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:12:46.279503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.279469 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:12:46.279503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.279492 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx"] Nov 02 22:12:46.288257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.288234 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd"] Nov 02 22:12:46.305925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.305899 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd"] Nov 02 22:12:46.306122 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.306098 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.312129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.312107 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:12:46.312892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.312631 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:12:46.313013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.312775 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:12:46.313165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.312851 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:12:46.320514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.320493 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx"] Nov 02 22:12:46.337923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.337892 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.338470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.338446 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.339506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.339480 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.339688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.339653 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.339778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.339749 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.339847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.339832 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.339948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.339933 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.340063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.340047 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.340133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.340121 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkxg5\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-kube-api-access-pkxg5\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.343075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.343041 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.352608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.352508 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx"] Nov 02 22:12:46.369423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.369381 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr"] Nov 02 22:12:46.388067 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.388024 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr"] Nov 02 22:12:46.388614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.388501 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.444994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.444876 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.445285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.445246 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.445417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.445297 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpf9\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-kube-api-access-5qpf9\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.445836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.445807 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbq5\" (UniqueName: \"kubernetes.io/projected/a8c986e4-40d6-4841-9e86-cef998f99160-kube-api-access-spbq5\") pod \"maas-api-revoke-sub-4e1e8d2b-vc9cd\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.445991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.445975 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.446104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.446091 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e3a1ba2-4zlvr\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.446235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.446219 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.446331 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.446318 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7ln\" (UniqueName: \"kubernetes.io/projected/c45e5db9-eea7-43cf-ba46-1595d9511182-kube-api-access-vv7ln\") pod \"maas-api-revoke-sub-1e3a1ba2-4zlvr\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.447760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.447734 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1e8d2b-vc9cd\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.447869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.447829 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.447869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.447862 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.447983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.447895 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.447983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.447931 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.447983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.447957 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkxg5\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-kube-api-access-pkxg5\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.448135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448001 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.448135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448034 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5h4b\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-kube-api-access-l5h4b\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.448135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448063 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1e8d2b-vc9cd\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.448135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448112 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.448314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448148 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bd580c-fb5e-448f-a476-ceb46734b65b-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.448314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448193 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.448314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448232 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.448314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448280 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.448314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448306 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.448553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448329 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e3a1ba2-4zlvr\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.448553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448355 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.448553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448386 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c857e-7bc5-481a-8296-3d393ba94437-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.448553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448447 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.448553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448482 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.448553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448544 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.448851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448576 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.448951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.448926 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-data\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.449064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.449044 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.449410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.449390 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-certs\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.449588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.449568 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.449715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.449700 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c857e-7bc5-481a-8296-3d393ba94437-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.450041 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.450023 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-credential-socket\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.450191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.450175 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.450976 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.450955 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istiod-ca-cert\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.477258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.476102 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-podinfo\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.477258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.476954 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-envoy\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.478263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.478237 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-token\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.479005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.478965 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkxg5\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-kube-api-access-pkxg5\") pod \"e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.552002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.551969 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552006 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7ln\" (UniqueName: \"kubernetes.io/projected/c45e5db9-eea7-43cf-ba46-1595d9511182-kube-api-access-vv7ln\") pod \"maas-api-revoke-sub-1e3a1ba2-4zlvr\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.552237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552028 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1e8d2b-vc9cd\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.552237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552051 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552088 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552113 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.552237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552153 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.552237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552186 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5h4b\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-kube-api-access-l5h4b\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552220 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1e8d2b-vc9cd\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.552614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552289 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bd580c-fb5e-448f-a476-ceb46734b65b-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552340 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552396 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.552614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552422 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.552614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552450 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e3a1ba2-4zlvr\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.552614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552476 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552509 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c857e-7bc5-481a-8296-3d393ba94437-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.552614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552518 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-credential-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552585 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.552988 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552666 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552988 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552761 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c857e-7bc5-481a-8296-3d393ba94437-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.552988 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552804 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.552988 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552827 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1e8d2b-vc9cd\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.552988 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552940 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.552988 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.552975 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpf9\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-kube-api-access-5qpf9\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.553284 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.553043 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbq5\" (UniqueName: \"kubernetes.io/projected/a8c986e4-40d6-4841-9e86-cef998f99160-kube-api-access-spbq5\") pod \"maas-api-revoke-sub-4e1e8d2b-vc9cd\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.553284 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.553077 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e3a1ba2-4zlvr\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.553284 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.553147 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-data\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.553438 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.553317 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bd580c-fb5e-448f-a476-ceb46734b65b-istiod-ca-cert\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.553498 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.553432 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-socket\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.553584 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.553551 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-credential-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.553905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.553881 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-data\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.554049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.554027 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c857e-7bc5-481a-8296-3d393ba94437-istiod-ca-cert\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.554131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.554111 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-certs\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.554376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.554356 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-certs\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.554563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.554540 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-socket\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.555492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.555466 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c857e-7bc5-481a-8296-3d393ba94437-istio-podinfo\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.555706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.555661 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e3a1ba2-4zlvr\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.556319 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.556291 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e3a1ba2-4zlvr\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.558623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.558601 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-envoy\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.566248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.566209 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-envoy\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.567640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.567612 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1e8d2b-vc9cd\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.568333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.568129 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-podinfo\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.571893 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.571867 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5h4b\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-kube-api-access-l5h4b\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.573350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.573227 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpf9\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-kube-api-access-5qpf9\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.574169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.574145 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-istio-token\") pod \"e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.576537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.576491 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-token\") pod \"e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.576982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.576943 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7ln\" (UniqueName: \"kubernetes.io/projected/c45e5db9-eea7-43cf-ba46-1595d9511182-kube-api-access-vv7ln\") pod \"maas-api-revoke-sub-1e3a1ba2-4zlvr\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.578322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.578297 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbq5\" (UniqueName: \"kubernetes.io/projected/a8c986e4-40d6-4841-9e86-cef998f99160-kube-api-access-spbq5\") pod \"maas-api-revoke-sub-4e1e8d2b-vc9cd\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.624059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.624021 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:46.644913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.644887 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:46.655051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.655026 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:46.676444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.676339 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:46.721391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.720552 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:46.977793 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:46.971524 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bd580c_fb5e_448f_a476_ceb46734b65b.slice/crio-f0937ff2d2e1aed3c25287c841a4e55e3a740f5e5021eb50a84240d5062992fd WatchSource:0}: Error finding container f0937ff2d2e1aed3c25287c841a4e55e3a740f5e5021eb50a84240d5062992fd: Status 404 returned error can't find the container with id f0937ff2d2e1aed3c25287c841a4e55e3a740f5e5021eb50a84240d5062992fd Nov 02 22:12:46.977793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.972591 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh"] Nov 02 22:12:46.977793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.974499 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:46.977793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.974585 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:46.977793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:46.974626 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:47.006939 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.006890 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd"] Nov 02 22:12:47.041894 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:47.038215 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7 WatchSource:0}: Error finding container aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7: Status 404 returned error can't find the container with id aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7 Nov 02 22:12:47.041894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.038402 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:47.041894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.038488 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:47.041894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.038534 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:47.041894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.041494 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx"] Nov 02 22:12:47.095947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.093491 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr"] Nov 02 22:12:47.112532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.112502 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx"] Nov 02 22:12:47.126847 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:47.126801 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b WatchSource:0}: Error finding container b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b: Status 404 returned error can't find the container with id b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b Nov 02 22:12:47.149602 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:12:47.149554 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47c857e_7bc5_481a_8296_3d393ba94437.slice/crio-b3bee52b66bfe922b6de2a7d0439ff1ef5df4e728b2e595fd59d7b881f2b8291 WatchSource:0}: Error finding container b3bee52b66bfe922b6de2a7d0439ff1ef5df4e728b2e595fd59d7b881f2b8291: Status 404 returned error can't find the container with id b3bee52b66bfe922b6de2a7d0439ff1ef5df4e728b2e595fd59d7b881f2b8291 Nov 02 22:12:47.154862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.154087 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:47.154862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.154175 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:47.154862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.154243 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:12:47.286796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.286352 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" event={"ID":"72bd580c-fb5e-448f-a476-ceb46734b65b","Type":"ContainerStarted","Data":"559f582933eaad7c2a09f4ab89d8caeab3e662637e27cf7f6fb7f4664a2576d7"} Nov 02 22:12:47.286796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.286425 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" event={"ID":"72bd580c-fb5e-448f-a476-ceb46734b65b","Type":"ContainerStarted","Data":"f0937ff2d2e1aed3c25287c841a4e55e3a740f5e5021eb50a84240d5062992fd"} Nov 02 22:12:47.290074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.289997 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" event={"ID":"c47c857e-7bc5-481a-8296-3d393ba94437","Type":"ContainerStarted","Data":"b3bee52b66bfe922b6de2a7d0439ff1ef5df4e728b2e595fd59d7b881f2b8291"} Nov 02 22:12:47.292692 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.292651 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" event={"ID":"908dd6a3-83b4-439d-a9e1-e76b8d8648e0","Type":"ContainerStarted","Data":"1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3"} Nov 02 22:12:47.292864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.292843 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" event={"ID":"908dd6a3-83b4-439d-a9e1-e76b8d8648e0","Type":"ContainerStarted","Data":"b6e1c9d5db4699f3570aafcfd623aa39f54e254fd8a5dc132b1a67101f23ec4d"} Nov 02 22:12:47.299016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.298991 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" event={"ID":"a8c986e4-40d6-4841-9e86-cef998f99160","Type":"ContainerStarted","Data":"aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7"} Nov 02 22:12:47.300976 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.300955 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" event={"ID":"c45e5db9-eea7-43cf-ba46-1595d9511182","Type":"ContainerStarted","Data":"b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b"} Nov 02 22:12:47.321038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.318082 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podStartSLOduration=3.318063116 podStartE2EDuration="3.318063116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:47.315123904 +0000 UTC m=+895.253237146" watchObservedRunningTime="2026-11-02 22:12:47.318063116 +0000 UTC m=+895.256176334" Nov 02 22:12:47.365035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.364001 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podStartSLOduration=3.363978534 podStartE2EDuration="3.363978534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:47.336931988 +0000 UTC m=+895.275045230" watchObservedRunningTime="2026-11-02 22:12:47.363978534 +0000 UTC m=+895.302091754" Nov 02 22:12:47.625401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.625076 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:47.645657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:47.645627 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:48.389699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.388148 4309 generic.go:356] "Generic (PLEG): container finished" podID="a8c986e4-40d6-4841-9e86-cef998f99160" containerID="e7b42638c8f2075d19738ccfabc09f8c1e7b010fc1b4ba5568f0c0b592a442f5" exitCode=0 Nov 02 22:12:48.389699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.388325 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" event={"ID":"a8c986e4-40d6-4841-9e86-cef998f99160","Type":"ContainerDied","Data":"e7b42638c8f2075d19738ccfabc09f8c1e7b010fc1b4ba5568f0c0b592a442f5"} Nov 02 22:12:48.405755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.400918 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" event={"ID":"c45e5db9-eea7-43cf-ba46-1595d9511182","Type":"ContainerStarted","Data":"eb790522427fb2f4f6b4180aa4645a5a443377ce9ce3aaf57278610e339c19c8"} Nov 02 22:12:48.569832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.543940 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" podStartSLOduration=3.543914034 podStartE2EDuration="3.543914034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:48.46205358 +0000 UTC m=+896.400166799" watchObservedRunningTime="2026-11-02 22:12:48.543914034 +0000 UTC m=+896.482027253" Nov 02 22:12:48.657645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.634114 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" event={"ID":"c47c857e-7bc5-481a-8296-3d393ba94437","Type":"ContainerStarted","Data":"f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914"} Nov 02 22:12:48.674984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.672390 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 22:12:48.674984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.672453 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 22:12:48.689561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.689463 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:48.770700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.706621 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podStartSLOduration=4.706599053 podStartE2EDuration="4.706599053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:12:48.658712767 +0000 UTC m=+896.596825988" watchObservedRunningTime="2026-11-02 22:12:48.706599053 +0000 UTC m=+896.644712273" Nov 02 22:12:48.770700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.764826 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:12:48.770700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.764889 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 22:12:48.770700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.767110 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 22:12:48.770700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:48.767158 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 22:12:49.629701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:49.629289 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:12:49.629701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:49.629357 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 22:12:49.721264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:49.712731 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 22:12:49.721264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:49.712887 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 22:12:49.746700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:49.731531 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 22:12:49.746700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:49.731589 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 22:12:49.829483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:49.829444 4309 generic.go:356] "Generic (PLEG): container finished" podID="c45e5db9-eea7-43cf-ba46-1595d9511182" containerID="eb790522427fb2f4f6b4180aa4645a5a443377ce9ce3aaf57278610e339c19c8" exitCode=0 Nov 02 22:12:49.830237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:49.830199 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" event={"ID":"c45e5db9-eea7-43cf-ba46-1595d9511182","Type":"ContainerDied","Data":"eb790522427fb2f4f6b4180aa4645a5a443377ce9ce3aaf57278610e339c19c8"} Nov 02 22:12:50.687928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:50.687819 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 22:12:51.191307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.188611 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 22:12:51.191307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:50.810834 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:12:51.191307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:50.779662 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:51.191307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:50.709945 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:12:51.191307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.189351 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 22:12:51.191307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:50.840966 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 22:12:51.191307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.189413 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 22:12:51.380630 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:12:51.380202 4309 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.166s" Nov 02 22:12:51.454934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.454871 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:51.584117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.584077 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:51.630964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.630582 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:12:51.630964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.630638 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 22:12:51.630964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.630661 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:51.680917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.663826 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 22:12:51.680917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.663914 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 22:12:51.694714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.681446 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:51.706705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.695768 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 22:12:51.706705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.695837 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 22:12:51.707452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.707075 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:51.771064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.771025 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-xks-ca\") pod \"a8c986e4-40d6-4841-9e86-cef998f99160\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " Nov 02 22:12:51.812738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.783341 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c986e4-40d6-4841-9e86-cef998f99160-kube-api-access-spbq5\" (UniqueName: \"kubernetes.io/projected/a8c986e4-40d6-4841-9e86-cef998f99160-kube-api-access-spbq5\") pod \"a8c986e4-40d6-4841-9e86-cef998f99160\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " Nov 02 22:12:51.812738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.783410 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-openshift-ca\") pod \"a8c986e4-40d6-4841-9e86-cef998f99160\" (UID: \"a8c986e4-40d6-4841-9e86-cef998f99160\") " Nov 02 22:12:51.904825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.836887 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-openshift-ca" pod "a8c986e4-40d6-4841-9e86-cef998f99160" (UID: "a8c986e4-40d6-4841-9e86-cef998f99160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:51.904825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.871641 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:51.904825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.878635 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c986e4-40d6-4841-9e86-cef998f99160-kube-api-access-spbq5" pod "a8c986e4-40d6-4841-9e86-cef998f99160" (UID: "a8c986e4-40d6-4841-9e86-cef998f99160"). InnerVolumeSpecName "kube-api-access-spbq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:51.926857 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.926822 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-xks-ca" pod "a8c986e4-40d6-4841-9e86-cef998f99160" (UID: "a8c986e4-40d6-4841-9e86-cef998f99160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:51.967913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.967871 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:12:51.978270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.978232 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c986e4-40d6-4841-9e86-cef998f99160-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:51.978270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:51.978272 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spbq5\" (UniqueName: \"kubernetes.io/projected/a8c986e4-40d6-4841-9e86-cef998f99160-kube-api-access-spbq5\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:52.283701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.232645 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:53.029733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.533037 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.598793 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.825954 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.827840 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.034748 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.945134 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.945099 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.842693 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.035064 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.842740 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.035104 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.983019 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" event={"ID":"a8c986e4-40d6-4841-9e86-cef998f99160","Type":"ContainerDied","Data":"aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7"} Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.035186 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.035249 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.035287 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.983275 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd" Nov 02 22:12:53.082700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:52.984419 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:53.180124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.180022 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-openshift-ca\") pod \"c45e5db9-eea7-43cf-ba46-1595d9511182\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " Nov 02 22:12:53.180465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.180438 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c45e5db9-eea7-43cf-ba46-1595d9511182-kube-api-access-vv7ln\" (UniqueName: \"kubernetes.io/projected/c45e5db9-eea7-43cf-ba46-1595d9511182-kube-api-access-vv7ln\") pod \"c45e5db9-eea7-43cf-ba46-1595d9511182\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " Nov 02 22:12:53.180776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.180754 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-xks-ca\") pod \"c45e5db9-eea7-43cf-ba46-1595d9511182\" (UID: \"c45e5db9-eea7-43cf-ba46-1595d9511182\") " Nov 02 22:12:53.182944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.182922 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-openshift-ca" pod "c45e5db9-eea7-43cf-ba46-1595d9511182" (UID: "c45e5db9-eea7-43cf-ba46-1595d9511182"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:12:53.302705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.302176 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:53.302705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.302413 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c45e5db9-eea7-43cf-ba46-1595d9511182-kube-api-access-vv7ln" pod "c45e5db9-eea7-43cf-ba46-1595d9511182" (UID: "c45e5db9-eea7-43cf-ba46-1595d9511182"). InnerVolumeSpecName "kube-api-access-vv7ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:12:53.449755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.415493 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv7ln\" (UniqueName: \"kubernetes.io/projected/c45e5db9-eea7-43cf-ba46-1595d9511182-kube-api-access-vv7ln\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:53.449755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.433739 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-xks-ca" pod "c45e5db9-eea7-43cf-ba46-1595d9511182" (UID: "c45e5db9-eea7-43cf-ba46-1595d9511182"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:12:53.516064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.509988 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:53.557868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.518604 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c45e5db9-eea7-43cf-ba46-1595d9511182-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:12:53.558086 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.518652 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:53.663504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.650829 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 22:12:53.663504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.650888 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 22:12:53.663504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.651449 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:12:53.663504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.651488 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 22:12:53.806077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.805969 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:53.809876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.809833 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 22:12:53.810072 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.810049 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 22:12:53.994112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:53.994065 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:54.085792 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.079812 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:54.085792 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.080365 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:12:54.161700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.136339 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:12:54.554283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.554078 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:12:54.560206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.162050 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:12:54.564753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.500140 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:54.564753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.162154 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:54.606720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.577923 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" Nov 02 22:12:54.637698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.629449 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 22:12:54.637698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.629517 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 22:12:54.645425 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.645390 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 22:12:54.645581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.645447 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 22:12:54.665516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.665479 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr" event={"ID":"c45e5db9-eea7-43cf-ba46-1595d9511182","Type":"ContainerDied","Data":"b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b"} Nov 02 22:12:54.665516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.665518 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b" Nov 02 22:12:54.665804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.665577 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:12:54.665804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.665628 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:12:54.665804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.665662 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:12:54.665804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.665710 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:12:54.665804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.665742 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:12:54.665804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.665777 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:12:54.676801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.676772 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 22:12:54.676955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.676818 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 22:12:54.772329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.771095 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:12:54.781523 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:54.781107 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:12:55.646319 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:55.646287 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 22:12:55.646827 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:55.646773 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 22:12:55.677400 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:55.677370 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 22:12:55.677700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:55.677650 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 22:12:56.323966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:56.323934 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:56.325953 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:56.325931 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:56.330585 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:56.330560 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:12:56.526482 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:12:56.526443 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache]" Nov 02 22:12:56.650661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:56.650572 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:56.651213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:56.651101 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:56.651948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:56.651929 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:12:56.681466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:56.681439 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:56.681798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:56.681781 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:56.682511 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:56.682490 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:12:58.177696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.177377 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:58.407824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.407776 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:58.417952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.417914 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:58.439290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.439197 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:58.583012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.582964 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:58.583826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.583798 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 02 22:12:58.617269 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.617214 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:58.670094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.670050 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:58.670984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.670953 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:58.882923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.882871 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:58.993825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.993777 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:58.994134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.994107 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:58.994606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:58.994577 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:12:59.192710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.192598 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:59.193709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.193666 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:12:59.484758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.474491 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:59.484758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.475105 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:12:59.484758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.476445 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:59.484758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.476722 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:12:59.484758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.476786 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:12:59.484758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.476843 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:12:59.656717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.632753 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:12:59.744763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.730931 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:12:59.779706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.732254 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:12:59.779706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.732284 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:12:59.781958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.732304 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:12:59.782170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.782143 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:12:59.782242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.782215 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:12:59.782297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.782249 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:12:59.921029 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.920987 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5"] Nov 02 22:12:59.933706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.925373 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:12:59.975701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.944776 4309 cpu_manager.go:414] "containerMap: removing container" podUID="a8c986e4-40d6-4841-9e86-cef998f99160" containerName="revoke-keys" Nov 02 22:12:59.975701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.944804 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c986e4-40d6-4841-9e86-cef998f99160" containerName="revoke-keys" Nov 02 22:12:59.975701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.944870 4309 cpu_manager.go:414] "containerMap: removing container" podUID="c45e5db9-eea7-43cf-ba46-1595d9511182" containerName="revoke-keys" Nov 02 22:12:59.975701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.944880 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c45e5db9-eea7-43cf-ba46-1595d9511182" containerName="revoke-keys" Nov 02 22:12:59.975701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.945124 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c986e4-40d6-4841-9e86-cef998f99160" containerName="revoke-keys" Nov 02 22:12:59.975701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.945147 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="c45e5db9-eea7-43cf-ba46-1595d9511182" containerName="revoke-keys" Nov 02 22:12:59.975701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:12:59.951246 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:13:00.002321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.001022 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.051910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.020832 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:00.051910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.033842 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:00.051910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.034214 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:13:00.051910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.034275 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:13:00.102767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.075291 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Nov 02 22:13:00.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.153753 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.153835 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.153867 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.153949 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.153975 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.154017 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9e0e9b-0636-4f92-9251-171ea770c528-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.154108 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.154139 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-484r7\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-kube-api-access-484r7\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.154698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.154409 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.210769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.210730 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:00.277990 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.226700 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:00.314699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.261181 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.438749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.438710 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.439103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.439077 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.439282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.439262 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.439545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.439516 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.455708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.454328 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.455708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.454402 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9e0e9b-0636-4f92-9251-171ea770c528-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.455708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.454503 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.455708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.454533 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-484r7\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-kube-api-access-484r7\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.456075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.455713 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9e0e9b-0636-4f92-9251-171ea770c528-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.456075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.456002 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.471835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.456499 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:00.480802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.480755 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:13:00.672770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.644812 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:00.918701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.912801 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:13:00.941177 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.919469 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:13:01.001701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.920865 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:01.002342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.002311 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:01.002521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.922526 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:01.002630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.944981 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn"] Nov 02 22:13:01.002730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.945037 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:01.002871 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.701365 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:01.151242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.021753 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:01.151242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:00.920911 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:13:01.151242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.024165 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:01.151242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.041519 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-484r7\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-kube-api-access-484r7\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:01.151242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.042040 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:01.151242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.077026 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:01.172100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.172064 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:01.251752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.233287 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.506978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.414946 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:01.782756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.643837 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:01.793979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.793938 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqb7f\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-kube-api-access-rqb7f\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.824228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.824192 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.834830 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.834791 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.871655 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.871621 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cae483b-0cfe-488a-a47b-ca268dd12374-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.871990 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.871970 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.872203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.872188 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.872510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.872493 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.876826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.839629 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5"] Nov 02 22:13:01.876826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.665218 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:01.877845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.877818 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.878084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.878048 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:01.903776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:01.839725 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:02.019706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.019646 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:02.087127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.086862 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:13:02.219702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.129382 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:02.770716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.171307 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqb7f\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-kube-api-access-rqb7f\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.818019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.817972 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.818378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.818337 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.818631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.818614 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cae483b-0cfe-488a-a47b-ca268dd12374-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.818805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.818788 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.818965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.818948 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.859067 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.853193 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.859067 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.853432 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.859067 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.853492 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.859067 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.850625 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:13:02.878525 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.360321 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Nov 02 22:13:02.892705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.883354 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.907874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.238150 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:02.930471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.884390 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-certs\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.930471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.836789 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:02.930471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.884745 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-data\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.930471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.821371 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-credential-socket\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:02.930471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.892940 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Nov 02 22:13:02.930915 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.239006 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:13:02.936242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.936196 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:03.071751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.317951 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:13:03.117064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.117024 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cae483b-0cfe-488a-a47b-ca268dd12374-istiod-ca-cert\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:03.119810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.358634 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:03.120568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:02.360271 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:13:03.180825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.137868 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:13:03.226175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.179904 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:03.331664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.331563 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:03.331969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.331925 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:03.377229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.355664 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-token\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:03.414314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.414222 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqb7f\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-kube-api-access-rqb7f\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:03.633249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.514463 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-podinfo\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:03.633249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.557308 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn"] Nov 02 22:13:03.633249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.557401 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:13:03.633249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.617030 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:13:03.633249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.623711 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:13:03.633249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.624345 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:13:03.633249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.624458 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:13:03.653698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.642332 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:13:03.682462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.676470 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-envoy\") pod \"e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:03.682462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.677024 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:13:03.682462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.681467 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:13:03.690413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.690386 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:13:03.702255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.702231 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:13:03.705359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.705336 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:03.717877 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.717848 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:13:03.976286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.963619 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5"] Nov 02 22:13:03.976286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.965072 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:03.976286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.965145 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:03.976286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:03.965178 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:04.077361 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:04.077324 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cae483b_0cfe_488a_a47b_ca268dd12374.slice/crio-f8aedd0fe16c22b834b08b0a9d36409582dc655c3c7471a832d44ce80ed31810 WatchSource:0}: Error finding container f8aedd0fe16c22b834b08b0a9d36409582dc655c3c7471a832d44ce80ed31810: Status 404 returned error can't find the container with id f8aedd0fe16c22b834b08b0a9d36409582dc655c3c7471a832d44ce80ed31810 Nov 02 22:13:04.079111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.079089 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn"] Nov 02 22:13:04.081064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.081032 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:04.081160 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.081097 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:04.081160 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.081127 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:04.225252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.207646 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94"] Nov 02 22:13:04.233593 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.233475 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.239215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.239194 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:13:04.239743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.239723 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:13:04.239879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.239836 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:13:04.239951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.239896 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:13:04.242390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.242364 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94"] Nov 02 22:13:04.336696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.334539 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktscm\" (UniqueName: \"kubernetes.io/projected/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-kube-api-access-ktscm\") pod \"maas-api-revoke-sub-43c7de6f-4lw94\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.336696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.334825 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43c7de6f-4lw94\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.336696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.334902 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43c7de6f-4lw94\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.345727 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:04.344012 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:04.345727 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:04.344152 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:04.436710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.436364 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktscm\" (UniqueName: \"kubernetes.io/projected/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-kube-api-access-ktscm\") pod \"maas-api-revoke-sub-43c7de6f-4lw94\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.437568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.436634 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43c7de6f-4lw94\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.437568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.437021 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43c7de6f-4lw94\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.438997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.438224 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43c7de6f-4lw94\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.449264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.444725 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2"] Nov 02 22:13:04.457213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.456251 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43c7de6f-4lw94\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.458626 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.458603 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.460027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.459998 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2"] Nov 02 22:13:04.463467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.463275 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktscm\" (UniqueName: \"kubernetes.io/projected/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-kube-api-access-ktscm\") pod \"maas-api-revoke-sub-43c7de6f-4lw94\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.542859 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.538956 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f0c6d19-m46m2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.542859 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.539016 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f0c6d19-m46m2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.542859 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.539278 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcw86\" (UniqueName: \"kubernetes.io/projected/7100abcf-75ec-4990-9ead-fc45594e86d2-kube-api-access-mcw86\") pod \"maas-api-revoke-sub-5f0c6d19-m46m2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.568164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.568127 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:04.641718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.640596 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcw86\" (UniqueName: \"kubernetes.io/projected/7100abcf-75ec-4990-9ead-fc45594e86d2-kube-api-access-mcw86\") pod \"maas-api-revoke-sub-5f0c6d19-m46m2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.641718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.640811 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f0c6d19-m46m2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.641718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.640852 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f0c6d19-m46m2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.648370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.646033 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f0c6d19-m46m2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.648370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.646862 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f0c6d19-m46m2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.654739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.651724 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcw86\" (UniqueName: \"kubernetes.io/projected/7100abcf-75ec-4990-9ead-fc45594e86d2-kube-api-access-mcw86\") pod \"maas-api-revoke-sub-5f0c6d19-m46m2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.660273 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.660249 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn"] Nov 02 22:13:04.668445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.668413 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn"] Nov 02 22:13:04.668576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.668529 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.745934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.745369 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jkvc\" (UniqueName: \"kubernetes.io/projected/84cd6da1-1462-443b-b0f8-957a1073a6ea-kube-api-access-7jkvc\") pod \"maas-api-revoke-sub-dc3d4caa-cr9xn\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.745934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.745525 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc3d4caa-cr9xn\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.745934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.745901 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc3d4caa-cr9xn\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.767355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.767321 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" event={"ID":"5e9e0e9b-0636-4f92-9251-171ea770c528","Type":"ContainerStarted","Data":"fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc"} Nov 02 22:13:04.767550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.767378 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" event={"ID":"5e9e0e9b-0636-4f92-9251-171ea770c528","Type":"ContainerStarted","Data":"9a5483f2ec3f78078dafbcdd488e5126c7291fb32aa7ffb8f27988ace26b50d2"} Nov 02 22:13:04.770669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.770639 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" event={"ID":"2cae483b-0cfe-488a-a47b-ca268dd12374","Type":"ContainerStarted","Data":"82f6313d8638ef8c69c98999c0a9f62f7b1171f143851368cca9548d420afc61"} Nov 02 22:13:04.770669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.770704 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" event={"ID":"2cae483b-0cfe-488a-a47b-ca268dd12374","Type":"ContainerStarted","Data":"f8aedd0fe16c22b834b08b0a9d36409582dc655c3c7471a832d44ce80ed31810"} Nov 02 22:13:04.789894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.789425 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" podStartSLOduration=5.789406991 podStartE2EDuration="5.789406991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:04.78697391 +0000 UTC m=+912.725087128" watchObservedRunningTime="2026-11-02 22:13:04.789406991 +0000 UTC m=+912.727520213" Nov 02 22:13:04.817128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.817074 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podStartSLOduration=5.81705903 podStartE2EDuration="5.81705903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:04.809437122 +0000 UTC m=+912.747550340" watchObservedRunningTime="2026-11-02 22:13:04.81705903 +0000 UTC m=+912.755172248" Nov 02 22:13:04.847234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.847199 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc3d4caa-cr9xn\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.847440 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.847423 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc3d4caa-cr9xn\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.847554 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.847534 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jkvc\" (UniqueName: \"kubernetes.io/projected/84cd6da1-1462-443b-b0f8-957a1073a6ea-kube-api-access-7jkvc\") pod \"maas-api-revoke-sub-dc3d4caa-cr9xn\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.848096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.848059 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc3d4caa-cr9xn\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.848604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.848577 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:04.850330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.850307 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc3d4caa-cr9xn\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.855499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.855480 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jkvc\" (UniqueName: \"kubernetes.io/projected/84cd6da1-1462-443b-b0f8-957a1073a6ea-kube-api-access-7jkvc\") pod \"maas-api-revoke-sub-dc3d4caa-cr9xn\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:04.962591 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:04.960858 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e8c7e9_d07a_43d5_b8ba_9b82ec7b9fd4.slice/crio-0b1d2237e4e383d2aa6bb9f6ff41d11d51031343bd25adb4b33621ef1bdfe946 WatchSource:0}: Error finding container 0b1d2237e4e383d2aa6bb9f6ff41d11d51031343bd25adb4b33621ef1bdfe946: Status 404 returned error can't find the container with id 0b1d2237e4e383d2aa6bb9f6ff41d11d51031343bd25adb4b33621ef1bdfe946 Nov 02 22:13:04.970984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.968840 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94"] Nov 02 22:13:05.027812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:04.995953 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:05.161695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.154626 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2"] Nov 02 22:13:05.231429 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:05.231389 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7100abcf_75ec_4990_9ead_fc45594e86d2.slice/crio-0b5a796965c8f141c8684276719a200bd71aee9fdc0588ba52027b150a564085 WatchSource:0}: Error finding container 0b5a796965c8f141c8684276719a200bd71aee9fdc0588ba52027b150a564085: Status 404 returned error can't find the container with id 0b5a796965c8f141c8684276719a200bd71aee9fdc0588ba52027b150a564085 Nov 02 22:13:05.361195 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:05.361160 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cd6da1_1462_443b_b0f8_957a1073a6ea.slice/crio-839e5e7cf10128085cf16f31a4392ba7aec00434bff7ef3e562256fe7064cd3e WatchSource:0}: Error finding container 839e5e7cf10128085cf16f31a4392ba7aec00434bff7ef3e562256fe7064cd3e: Status 404 returned error can't find the container with id 839e5e7cf10128085cf16f31a4392ba7aec00434bff7ef3e562256fe7064cd3e Nov 02 22:13:05.361998 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.361974 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn"] Nov 02 22:13:05.548776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.548667 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7"] Nov 02 22:13:05.559979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.559949 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.562153 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.562123 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7"] Nov 02 22:13:05.732047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.732002 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9192c071-8968-4340-9e0b-65231b50255f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b895d99d-p5xd7\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.737586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.737551 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9192c071-8968-4340-9e0b-65231b50255f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b895d99d-p5xd7\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.756706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.746195 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6577\" (UniqueName: \"kubernetes.io/projected/9192c071-8968-4340-9e0b-65231b50255f-kube-api-access-v6577\") pod \"maas-api-revoke-sub-b895d99d-p5xd7\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.757644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.741491 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:05.757644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.757100 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:05.757644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.740800 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 22:13:05.757644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.757166 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 22:13:05.757644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.750907 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 22:13:05.757644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.757590 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 22:13:05.842452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.842394 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" event={"ID":"7100abcf-75ec-4990-9ead-fc45594e86d2","Type":"ContainerStarted","Data":"0b5a796965c8f141c8684276719a200bd71aee9fdc0588ba52027b150a564085"} Nov 02 22:13:05.845000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.844953 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" event={"ID":"84cd6da1-1462-443b-b0f8-957a1073a6ea","Type":"ContainerStarted","Data":"839e5e7cf10128085cf16f31a4392ba7aec00434bff7ef3e562256fe7064cd3e"} Nov 02 22:13:05.860705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.859305 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" event={"ID":"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4","Type":"ContainerStarted","Data":"fbc504eda0a6bcf369599b870d1a7a8c9e7de2178819633571e0fd2ce4b98972"} Nov 02 22:13:05.860705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.859405 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" event={"ID":"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4","Type":"ContainerStarted","Data":"0b1d2237e4e383d2aa6bb9f6ff41d11d51031343bd25adb4b33621ef1bdfe946"} Nov 02 22:13:05.861815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.861785 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9192c071-8968-4340-9e0b-65231b50255f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b895d99d-p5xd7\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.862125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.862103 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9192c071-8968-4340-9e0b-65231b50255f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b895d99d-p5xd7\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.862601 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.862580 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6577\" (UniqueName: \"kubernetes.io/projected/9192c071-8968-4340-9e0b-65231b50255f-kube-api-access-v6577\") pod \"maas-api-revoke-sub-b895d99d-p5xd7\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.863970 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.863946 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9192c071-8968-4340-9e0b-65231b50255f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b895d99d-p5xd7\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.868577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.868555 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9192c071-8968-4340-9e0b-65231b50255f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b895d99d-p5xd7\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.871884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:05.871850 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6577\" (UniqueName: \"kubernetes.io/projected/9192c071-8968-4340-9e0b-65231b50255f-kube-api-access-v6577\") pod \"maas-api-revoke-sub-b895d99d-p5xd7\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:05.945070 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:05.945021 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:06.065959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.065935 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:06.664651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.658503 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" podStartSLOduration=2.658481334 podStartE2EDuration="2.658481334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:05.878752841 +0000 UTC m=+913.816866062" watchObservedRunningTime="2026-11-02 22:13:06.658481334 +0000 UTC m=+914.596594553" Nov 02 22:13:06.664651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.659375 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7"] Nov 02 22:13:06.670575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.669752 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 22:13:06.670575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.669807 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 22:13:06.681579 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:06.681293 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9192c071_8968_4340_9e0b_65231b50255f.slice/crio-9d205cdb5aa40224dd07a3a58c2c89e0176ae6565af0a4c15d96811f61e1d098\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:06.739414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.739345 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 22:13:06.739662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.739601 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 22:13:06.867984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.867950 4309 generic.go:356] "Generic (PLEG): container finished" podID="13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4" containerID="fbc504eda0a6bcf369599b870d1a7a8c9e7de2178819633571e0fd2ce4b98972" exitCode=0 Nov 02 22:13:06.868298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.868020 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" event={"ID":"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4","Type":"ContainerDied","Data":"fbc504eda0a6bcf369599b870d1a7a8c9e7de2178819633571e0fd2ce4b98972"} Nov 02 22:13:06.870281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.870258 4309 generic.go:356] "Generic (PLEG): container finished" podID="7100abcf-75ec-4990-9ead-fc45594e86d2" containerID="a1b6aa19aa6a0d5505110c0d70579ea3ff32fc7b8c0dd04c28cc9acba6616b51" exitCode=0 Nov 02 22:13:06.870374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.870326 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" event={"ID":"7100abcf-75ec-4990-9ead-fc45594e86d2","Type":"ContainerDied","Data":"a1b6aa19aa6a0d5505110c0d70579ea3ff32fc7b8c0dd04c28cc9acba6616b51"} Nov 02 22:13:06.871973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.871948 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" event={"ID":"9192c071-8968-4340-9e0b-65231b50255f","Type":"ContainerStarted","Data":"9d205cdb5aa40224dd07a3a58c2c89e0176ae6565af0a4c15d96811f61e1d098"} Nov 02 22:13:06.873732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.873710 4309 generic.go:356] "Generic (PLEG): container finished" podID="84cd6da1-1462-443b-b0f8-957a1073a6ea" containerID="a2bccd24195cd9eecfdd94f12bb3c5a5a15d38c060e19dbf18990448ca8cd086" exitCode=0 Nov 02 22:13:06.873990 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:06.873778 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" event={"ID":"84cd6da1-1462-443b-b0f8-957a1073a6ea","Type":"ContainerDied","Data":"a2bccd24195cd9eecfdd94f12bb3c5a5a15d38c060e19dbf18990448ca8cd086"} Nov 02 22:13:07.482724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.481798 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj"] Nov 02 22:13:07.565700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.564516 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.572417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.572378 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj"] Nov 02 22:13:07.668315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.668268 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 22:13:07.668561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.668513 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 22:13:07.671965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.671937 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96s8b\" (UniqueName: \"kubernetes.io/projected/63b0299b-2bde-4fa6-a626-81264f95ea04-kube-api-access-96s8b\") pod \"maas-api-revoke-sub-ee61efa0-728sj\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.672067 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.672033 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee61efa0-728sj\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.672336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.672316 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee61efa0-728sj\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.741764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.732913 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 22:13:07.741764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.732966 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 22:13:07.773857 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.773811 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee61efa0-728sj\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.774024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.774000 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96s8b\" (UniqueName: \"kubernetes.io/projected/63b0299b-2bde-4fa6-a626-81264f95ea04-kube-api-access-96s8b\") pod \"maas-api-revoke-sub-ee61efa0-728sj\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.774118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.774100 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee61efa0-728sj\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.774759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.774733 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee61efa0-728sj\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.777071 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.777052 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee61efa0-728sj\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.817699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.817081 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96s8b\" (UniqueName: \"kubernetes.io/projected/63b0299b-2bde-4fa6-a626-81264f95ea04-kube-api-access-96s8b\") pod \"maas-api-revoke-sub-ee61efa0-728sj\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:07.956772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.953379 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" event={"ID":"9192c071-8968-4340-9e0b-65231b50255f","Type":"ContainerStarted","Data":"9d2b4937428d2a3d1613f15162f974b2612d8270559d0813775edf8f69f3f022"} Nov 02 22:13:07.977932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:07.977892 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:08.068134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.068078 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" podStartSLOduration=3.068058412 podStartE2EDuration="3.068058412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:07.990060078 +0000 UTC m=+915.928173297" watchObservedRunningTime="2026-11-02 22:13:08.068058412 +0000 UTC m=+916.006171631" Nov 02 22:13:08.735761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.683140 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 22:13:08.735761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.683199 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 22:13:08.735761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.718173 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 22:13:08.735761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.718228 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 22:13:08.737106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.736597 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:08.870710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.869636 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7100abcf-75ec-4990-9ead-fc45594e86d2-kube-api-access-mcw86\" (UniqueName: \"kubernetes.io/projected/7100abcf-75ec-4990-9ead-fc45594e86d2-kube-api-access-mcw86\") pod \"7100abcf-75ec-4990-9ead-fc45594e86d2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " Nov 02 22:13:08.870710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.869768 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-openshift-ca\") pod \"7100abcf-75ec-4990-9ead-fc45594e86d2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " Nov 02 22:13:08.870710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.869865 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-xks-ca\") pod \"7100abcf-75ec-4990-9ead-fc45594e86d2\" (UID: \"7100abcf-75ec-4990-9ead-fc45594e86d2\") " Nov 02 22:13:08.875766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.873993 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-openshift-ca" pod "7100abcf-75ec-4990-9ead-fc45594e86d2" (UID: "7100abcf-75ec-4990-9ead-fc45594e86d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:08.881646 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.880245 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7100abcf-75ec-4990-9ead-fc45594e86d2-kube-api-access-mcw86" pod "7100abcf-75ec-4990-9ead-fc45594e86d2" (UID: "7100abcf-75ec-4990-9ead-fc45594e86d2"). InnerVolumeSpecName "kube-api-access-mcw86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:08.944532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.944200 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-xks-ca" pod "7100abcf-75ec-4990-9ead-fc45594e86d2" (UID: "7100abcf-75ec-4990-9ead-fc45594e86d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:08.972963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.972860 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcw86\" (UniqueName: \"kubernetes.io/projected/7100abcf-75ec-4990-9ead-fc45594e86d2-kube-api-access-mcw86\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:08.972963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.972893 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:08.972963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.972910 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7100abcf-75ec-4990-9ead-fc45594e86d2-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:08.973780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:08.973699 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:09.070199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.070160 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" event={"ID":"84cd6da1-1462-443b-b0f8-957a1073a6ea","Type":"ContainerDied","Data":"839e5e7cf10128085cf16f31a4392ba7aec00434bff7ef3e562256fe7064cd3e"} Nov 02 22:13:09.070875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.070830 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="839e5e7cf10128085cf16f31a4392ba7aec00434bff7ef3e562256fe7064cd3e" Nov 02 22:13:09.070875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.070505 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn" Nov 02 22:13:09.072711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.072660 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" event={"ID":"7100abcf-75ec-4990-9ead-fc45594e86d2","Type":"ContainerDied","Data":"0b5a796965c8f141c8684276719a200bd71aee9fdc0588ba52027b150a564085"} Nov 02 22:13:09.072837 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.072733 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b5a796965c8f141c8684276719a200bd71aee9fdc0588ba52027b150a564085" Nov 02 22:13:09.074585 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.074564 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-xks-ca\") pod \"84cd6da1-1462-443b-b0f8-957a1073a6ea\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " Nov 02 22:13:09.074856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.074835 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-openshift-ca\") pod \"84cd6da1-1462-443b-b0f8-957a1073a6ea\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " Nov 02 22:13:09.075227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.075211 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84cd6da1-1462-443b-b0f8-957a1073a6ea-kube-api-access-7jkvc\" (UniqueName: \"kubernetes.io/projected/84cd6da1-1462-443b-b0f8-957a1073a6ea-kube-api-access-7jkvc\") pod \"84cd6da1-1462-443b-b0f8-957a1073a6ea\" (UID: \"84cd6da1-1462-443b-b0f8-957a1073a6ea\") " Nov 02 22:13:09.075766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.075745 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2" Nov 02 22:13:09.079297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.079275 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-openshift-ca" pod "84cd6da1-1462-443b-b0f8-957a1073a6ea" (UID: "84cd6da1-1462-443b-b0f8-957a1073a6ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:09.080323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.080299 4309 generic.go:356] "Generic (PLEG): container finished" podID="9192c071-8968-4340-9e0b-65231b50255f" containerID="9d2b4937428d2a3d1613f15162f974b2612d8270559d0813775edf8f69f3f022" exitCode=0 Nov 02 22:13:09.080519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.080505 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" event={"ID":"9192c071-8968-4340-9e0b-65231b50255f","Type":"ContainerDied","Data":"9d2b4937428d2a3d1613f15162f974b2612d8270559d0813775edf8f69f3f022"} Nov 02 22:13:09.112699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.109013 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84cd6da1-1462-443b-b0f8-957a1073a6ea-kube-api-access-7jkvc" pod "84cd6da1-1462-443b-b0f8-957a1073a6ea" (UID: "84cd6da1-1462-443b-b0f8-957a1073a6ea"). InnerVolumeSpecName "kube-api-access-7jkvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:09.118724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.112852 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-xks-ca" pod "84cd6da1-1462-443b-b0f8-957a1073a6ea" (UID: "84cd6da1-1462-443b-b0f8-957a1073a6ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:09.180136 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.180026 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:09.202297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.202265 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84cd6da1-1462-443b-b0f8-957a1073a6ea-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:09.202518 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.202501 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jkvc\" (UniqueName: \"kubernetes.io/projected/84cd6da1-1462-443b-b0f8-957a1073a6ea-kube-api-access-7jkvc\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:09.258099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.258069 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:09.323770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.310423 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-xks-ca\") pod \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " Nov 02 22:13:09.347703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.324126 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-openshift-ca\") pod \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " Nov 02 22:13:09.348251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.348214 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-kube-api-access-ktscm\" (UniqueName: \"kubernetes.io/projected/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-kube-api-access-ktscm\") pod \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\" (UID: \"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4\") " Nov 02 22:13:09.444952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.444909 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-openshift-ca" pod "13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4" (UID: "13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:09.469920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.469891 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:09.558288 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.557051 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-xks-ca" pod "13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4" (UID: "13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:09.563505 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:09.563481 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b0299b_2bde_4fa6_a626_81264f95ea04.slice/crio-c8afa4d1c39d932a26c1a870e3ad02fbf137618235698a7b7a13a3436453d6de WatchSource:0}: Error finding container c8afa4d1c39d932a26c1a870e3ad02fbf137618235698a7b7a13a3436453d6de: Status 404 returned error can't find the container with id c8afa4d1c39d932a26c1a870e3ad02fbf137618235698a7b7a13a3436453d6de Nov 02 22:13:09.564035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.564002 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-kube-api-access-ktscm" pod "13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4" (UID: "13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4"). InnerVolumeSpecName "kube-api-access-ktscm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:09.581728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.575309 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:09.581728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.575338 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktscm\" (UniqueName: \"kubernetes.io/projected/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4-kube-api-access-ktscm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:09.662299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.662257 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj"] Nov 02 22:13:09.669084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.668871 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 22:13:09.669607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.669577 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 22:13:09.732714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.730734 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 22:13:09.732714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:09.730804 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 22:13:10.165100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.165062 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" Nov 02 22:13:10.165621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.165070 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94" event={"ID":"13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4","Type":"ContainerDied","Data":"0b1d2237e4e383d2aa6bb9f6ff41d11d51031343bd25adb4b33621ef1bdfe946"} Nov 02 22:13:10.165621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.165198 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b1d2237e4e383d2aa6bb9f6ff41d11d51031343bd25adb4b33621ef1bdfe946" Nov 02 22:13:10.166926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.166897 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" event={"ID":"63b0299b-2bde-4fa6-a626-81264f95ea04","Type":"ContainerStarted","Data":"c8afa4d1c39d932a26c1a870e3ad02fbf137618235698a7b7a13a3436453d6de"} Nov 02 22:13:10.557589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.557567 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:10.647753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.647563 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9192c071-8968-4340-9e0b-65231b50255f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9192c071-8968-4340-9e0b-65231b50255f-maas-api-openshift-ca\") pod \"9192c071-8968-4340-9e0b-65231b50255f\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " Nov 02 22:13:10.647753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.647715 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9192c071-8968-4340-9e0b-65231b50255f-kube-api-access-v6577\" (UniqueName: \"kubernetes.io/projected/9192c071-8968-4340-9e0b-65231b50255f-kube-api-access-v6577\") pod \"9192c071-8968-4340-9e0b-65231b50255f\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " Nov 02 22:13:10.648695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.648174 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9192c071-8968-4340-9e0b-65231b50255f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9192c071-8968-4340-9e0b-65231b50255f-maas-api-xks-ca\") pod \"9192c071-8968-4340-9e0b-65231b50255f\" (UID: \"9192c071-8968-4340-9e0b-65231b50255f\") " Nov 02 22:13:10.653183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.648916 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9192c071-8968-4340-9e0b-65231b50255f-maas-api-openshift-ca" pod "9192c071-8968-4340-9e0b-65231b50255f" (UID: "9192c071-8968-4340-9e0b-65231b50255f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:10.653183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.649204 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9192c071-8968-4340-9e0b-65231b50255f-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:10.658736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.656447 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9192c071-8968-4340-9e0b-65231b50255f-kube-api-access-v6577" pod "9192c071-8968-4340-9e0b-65231b50255f" (UID: "9192c071-8968-4340-9e0b-65231b50255f"). InnerVolumeSpecName "kube-api-access-v6577". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:10.659618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.659328 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9192c071-8968-4340-9e0b-65231b50255f-maas-api-xks-ca" pod "9192c071-8968-4340-9e0b-65231b50255f" (UID: "9192c071-8968-4340-9e0b-65231b50255f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:10.667921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.665786 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 02 22:13:10.667921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.665859 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 02 22:13:10.732901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.732864 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 22:13:10.733141 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.733115 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 22:13:10.753740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.753662 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6577\" (UniqueName: \"kubernetes.io/projected/9192c071-8968-4340-9e0b-65231b50255f-kube-api-access-v6577\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:10.753740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:10.753727 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9192c071-8968-4340-9e0b-65231b50255f-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:11.177500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:11.177407 4309 generic.go:356] "Generic (PLEG): container finished" podID="63b0299b-2bde-4fa6-a626-81264f95ea04" containerID="2ecf5a08ef895c52b8c093ca7a943c1454852b97dd3ee592d07bc5ff057babc6" exitCode=0 Nov 02 22:13:11.178001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:11.177664 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" event={"ID":"63b0299b-2bde-4fa6-a626-81264f95ea04","Type":"ContainerDied","Data":"2ecf5a08ef895c52b8c093ca7a943c1454852b97dd3ee592d07bc5ff057babc6"} Nov 02 22:13:11.179339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:11.179312 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" Nov 02 22:13:11.179451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:11.179333 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7" event={"ID":"9192c071-8968-4340-9e0b-65231b50255f","Type":"ContainerDied","Data":"9d205cdb5aa40224dd07a3a58c2c89e0176ae6565af0a4c15d96811f61e1d098"} Nov 02 22:13:11.179451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:11.179382 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d205cdb5aa40224dd07a3a58c2c89e0176ae6565af0a4c15d96811f61e1d098" Nov 02 22:13:11.636164 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:11.636122 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:13:11.726699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:11.719340 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 02 22:13:11.726699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:11.719397 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 02 22:13:12.166006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.165974 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:12.166806 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.166771 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:12.167482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.167465 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:13:12.521699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.519264 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:12.555700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.554763 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-openshift-ca\") pod \"63b0299b-2bde-4fa6-a626-81264f95ea04\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " Nov 02 22:13:12.555957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.555924 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b0299b-2bde-4fa6-a626-81264f95ea04-kube-api-access-96s8b\" (UniqueName: \"kubernetes.io/projected/63b0299b-2bde-4fa6-a626-81264f95ea04-kube-api-access-96s8b\") pod \"63b0299b-2bde-4fa6-a626-81264f95ea04\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " Nov 02 22:13:12.556065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.556047 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-xks-ca\") pod \"63b0299b-2bde-4fa6-a626-81264f95ea04\" (UID: \"63b0299b-2bde-4fa6-a626-81264f95ea04\") " Nov 02 22:13:12.561555 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.559810 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-openshift-ca" pod "63b0299b-2bde-4fa6-a626-81264f95ea04" (UID: "63b0299b-2bde-4fa6-a626-81264f95ea04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:12.561796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.561755 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-xks-ca" pod "63b0299b-2bde-4fa6-a626-81264f95ea04" (UID: "63b0299b-2bde-4fa6-a626-81264f95ea04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:12.562619 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.562588 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b0299b-2bde-4fa6-a626-81264f95ea04-kube-api-access-96s8b" pod "63b0299b-2bde-4fa6-a626-81264f95ea04" (UID: "63b0299b-2bde-4fa6-a626-81264f95ea04"). InnerVolumeSpecName "kube-api-access-96s8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:12.658796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.658759 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:12.658796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.658788 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96s8b\" (UniqueName: \"kubernetes.io/projected/63b0299b-2bde-4fa6-a626-81264f95ea04-kube-api-access-96s8b\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:12.658991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.658806 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0299b-2bde-4fa6-a626-81264f95ea04-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:12.828392 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.828355 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:12.829168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.829139 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:12.830119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:12.830094 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:13:13.252748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:13.252615 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" Nov 02 22:13:13.252748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:13.252611 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj" event={"ID":"63b0299b-2bde-4fa6-a626-81264f95ea04","Type":"ContainerDied","Data":"c8afa4d1c39d932a26c1a870e3ad02fbf137618235698a7b7a13a3436453d6de"} Nov 02 22:13:13.252748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:13.252716 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8afa4d1c39d932a26c1a870e3ad02fbf137618235698a7b7a13a3436453d6de" Nov 02 22:13:14.105575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.105537 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2"] Nov 02 22:13:14.106295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106279 4309 cpu_manager.go:414] "containerMap: removing container" podUID="63b0299b-2bde-4fa6-a626-81264f95ea04" containerName="revoke-keys" Nov 02 22:13:14.106295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106295 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b0299b-2bde-4fa6-a626-81264f95ea04" containerName="revoke-keys" Nov 02 22:13:14.106409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106308 4309 cpu_manager.go:414] "containerMap: removing container" podUID="9192c071-8968-4340-9e0b-65231b50255f" containerName="revoke-keys" Nov 02 22:13:14.106409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106315 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9192c071-8968-4340-9e0b-65231b50255f" containerName="revoke-keys" Nov 02 22:13:14.106409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106328 4309 cpu_manager.go:414] "containerMap: removing container" podUID="13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4" containerName="revoke-keys" Nov 02 22:13:14.106409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106333 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4" containerName="revoke-keys" Nov 02 22:13:14.106409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106346 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7100abcf-75ec-4990-9ead-fc45594e86d2" containerName="revoke-keys" Nov 02 22:13:14.106409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106351 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7100abcf-75ec-4990-9ead-fc45594e86d2" containerName="revoke-keys" Nov 02 22:13:14.106409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106371 4309 cpu_manager.go:414] "containerMap: removing container" podUID="84cd6da1-1462-443b-b0f8-957a1073a6ea" containerName="revoke-keys" Nov 02 22:13:14.106409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106376 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84cd6da1-1462-443b-b0f8-957a1073a6ea" containerName="revoke-keys" Nov 02 22:13:14.106844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106483 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b0299b-2bde-4fa6-a626-81264f95ea04" containerName="revoke-keys" Nov 02 22:13:14.106844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106495 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="9192c071-8968-4340-9e0b-65231b50255f" containerName="revoke-keys" Nov 02 22:13:14.106844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106502 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4" containerName="revoke-keys" Nov 02 22:13:14.106844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106509 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7100abcf-75ec-4990-9ead-fc45594e86d2" containerName="revoke-keys" Nov 02 22:13:14.106844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.106519 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="84cd6da1-1462-443b-b0f8-957a1073a6ea" containerName="revoke-keys" Nov 02 22:13:14.112850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.110802 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.116320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.116288 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:13:14.117730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.117663 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:13:14.117837 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.117768 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:13:14.117896 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.117879 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:13:14.119658 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.119635 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2"] Nov 02 22:13:14.184260 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.184228 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb07e0a4-986b2\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.184466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.184397 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mcxs\" (UniqueName: \"kubernetes.io/projected/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-kube-api-access-2mcxs\") pod \"maas-api-revoke-sub-bb07e0a4-986b2\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.184466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.184456 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb07e0a4-986b2\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.285945 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.285912 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb07e0a4-986b2\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.286162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.286141 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mcxs\" (UniqueName: \"kubernetes.io/projected/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-kube-api-access-2mcxs\") pod \"maas-api-revoke-sub-bb07e0a4-986b2\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.286272 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.286244 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb07e0a4-986b2\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.287021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.286995 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb07e0a4-986b2\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.288553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.288529 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb07e0a4-986b2\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.293657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.293634 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mcxs\" (UniqueName: \"kubernetes.io/projected/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-kube-api-access-2mcxs\") pod \"maas-api-revoke-sub-bb07e0a4-986b2\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.436747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.433044 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:14.600696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.597587 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c"] Nov 02 22:13:14.608968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.608941 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:14.655727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.654866 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c"] Nov 02 22:13:14.834575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.812361 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:14.834575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.812416 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:14.834575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.812557 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5fq\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-kube-api-access-ww5fq\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:14.834575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.812649 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:14.834575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.812755 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35ad342-26b9-43a9-a327-02309df9f1dc-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:14.834575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.812902 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:14.834575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.812941 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:14.834575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.813001 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:14.834575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:14.813056 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.052544 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35ad342-26b9-43a9-a327-02309df9f1dc-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.052713 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.052782 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.052880 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.052929 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.053025 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.053054 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.053162 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5fq\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-kube-api-access-ww5fq\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.053252 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.055910 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35ad342-26b9-43a9-a327-02309df9f1dc-istiod-ca-cert\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.056222 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-credential-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.056744 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-data\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.086756 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-socket\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.098752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.087074 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-certs\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.375372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.347614 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-envoy\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.375372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.367055 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5fq\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-kube-api-access-ww5fq\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.395701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.376635 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-podinfo\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.485699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.457214 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-token\") pod \"e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.676343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.676312 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:15.719622 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.719588 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2"] Nov 02 22:13:15.777221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.777177 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" event={"ID":"b1ee6f97-4e20-4fc7-a7fb-bc1288836488","Type":"ContainerStarted","Data":"b3721de76edda1bf0a9dcf9138925038c79b29335ac9a418e60317b6aa4bde06"} Nov 02 22:13:15.804699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:15.803776 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:16.025757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.023591 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:16.044474 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.043822 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:16.048550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.043860 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:16.049997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.043888 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:16.397700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.366361 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:16.642711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.641436 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:13:16.647695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.645788 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:16.698372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.694839 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:16.740030 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.737999 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:13:16.867927 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:16.867885 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:17.217057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.023897 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:13:17.226038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.049775 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:17.230790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.049814 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:17.231219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.049838 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:17.231818 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.051424 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:17.232098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.051455 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:17.374092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.165795 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:13:17.481159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.219871 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:13:17.481820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.481791 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:13:17.489854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.489830 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:13:17.490026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.490011 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:13:17.517700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.490221 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:13:17.517985 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.517954 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" event={"ID":"b1ee6f97-4e20-4fc7-a7fb-bc1288836488","Type":"ContainerStarted","Data":"11d03f3f5291cbc74d29d186dab6a86ff55a581f1cede96a4ac3bc6e9520ef72"} Nov 02 22:13:17.518132 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.512055 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" podStartSLOduration=3.5120325599999997 podStartE2EDuration="3.51203256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:17.364355182 +0000 UTC m=+925.302468402" watchObservedRunningTime="2026-11-02 22:13:17.51203256 +0000 UTC m=+925.450145780" Nov 02 22:13:17.689773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.659796 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:17.716825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.711782 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:17.754728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.752526 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:13:17.754728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.752601 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:13:17.754728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.711821 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:13:17.754728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.753752 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:13:17.784130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.755759 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:13:17.821709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.808508 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:13:17.956184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.955285 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:17.956184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.955419 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:17.956184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:17.955497 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:18.238762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.237765 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:18.244801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.240268 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:18.244801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.240497 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:13:18.244801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.240668 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:13:18.244801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.240854 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:13:18.276264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.276221 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:18.276560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.276527 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:18.410925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.357034 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:18.410925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.357096 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:18.410925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.394888 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:13:18.531757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.492416 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:13:18.645699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.642073 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:18.750861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.750818 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:13:18.782539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.782442 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:13:18.794594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.784754 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:13:18.864844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:18.836009 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:13:19.118022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.117962 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:19.298698 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:19.265507 4309 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.026s" Nov 02 22:13:19.417958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.417869 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:13:19.420815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.420770 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:13:19.430827 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.301755 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:19.440093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.302398 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:13:19.526729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.302428 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:19.527294 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.302459 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:19.527782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.417079 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:19.555962 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.525005 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:19.737232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.725468 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" event={"ID":"c35ad342-26b9-43a9-a327-02309df9f1dc","Type":"ContainerStarted","Data":"23fa0d28a17bf98c1ae812e597213871ca35ba9bc20a068591c31c1ce58c2608"} Nov 02 22:13:19.771739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.751628 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:13:19.830727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.302478 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:13:19.846188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.846144 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:13:20.126166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.876182 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:13:20.126699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.879335 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:20.127779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.927061 4309 generic.go:356] "Generic (PLEG): container finished" podID="b1ee6f97-4e20-4fc7-a7fb-bc1288836488" containerID="11d03f3f5291cbc74d29d186dab6a86ff55a581f1cede96a4ac3bc6e9520ef72" exitCode=0 Nov 02 22:13:20.133946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:19.927092 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" event={"ID":"b1ee6f97-4e20-4fc7-a7fb-bc1288836488","Type":"ContainerDied","Data":"11d03f3f5291cbc74d29d186dab6a86ff55a581f1cede96a4ac3bc6e9520ef72"} Nov 02 22:13:20.145700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:20.087252 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Nov 02 22:13:20.219351 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:20.219280 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:20.703702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:20.703619 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:20.898698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:20.898191 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:20.951000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:20.946235 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:13:21.203824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.163353 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:21.203824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.164580 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:13:21.203824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.165336 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:13:21.305620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.301116 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:13:21.305620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.301867 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:21.305620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.301995 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:13:21.305620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.302091 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:13:21.348159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.348109 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:21.706119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.706077 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:21.905699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.898454 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:21.947862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:21.947815 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:22.226698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.225964 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": context deadline exceeded" Nov 02 22:13:22.226698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.226276 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:22.226698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.226598 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:13:22.226991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.226936 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:13:22.228838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.228813 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" podUID="774397e4-c8ee-4734-8f5a-7befd9b4d9dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 02 22:13:22.315252 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:22.315211 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:22.326779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.323632 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:22.326779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.323930 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:13:22.341695 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:22.341626 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ee6f97_4e20_4fc7_a7fb_bc1288836488.slice/crio-conmon-11d03f3f5291cbc74d29d186dab6a86ff55a581f1cede96a4ac3bc6e9520ef72.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:22.388892 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:22.388558 4309 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.202s" Nov 02 22:13:22.388892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.388625 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:13:22.388892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.388685 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:13:22.388892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.388718 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:13:22.388892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.388737 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c"] Nov 02 22:13:22.389230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.388945 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:13:22.389230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.388983 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:13:22.456535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.456504 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:13:22.457423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.457362 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:13:22.457593 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.457516 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:13:22.501863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.500424 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:13:22.608651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.608571 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:13:22.631522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.631490 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:13:22.647754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.647715 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2"] Nov 02 22:13:22.658942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.658914 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.662149 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.662127 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2"] Nov 02 22:13:22.735253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.720726 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5cc3752-xt7z2\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.735253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.721214 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnh7\" (UniqueName: \"kubernetes.io/projected/df0a2d65-7bc3-4760-90c6-0717a8a70502-kube-api-access-jhnh7\") pod \"maas-api-revoke-sub-b5cc3752-xt7z2\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.735253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.721477 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5cc3752-xt7z2\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.823963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.823926 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5cc3752-xt7z2\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.824133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.823998 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5cc3752-xt7z2\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.825028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.824900 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnh7\" (UniqueName: \"kubernetes.io/projected/df0a2d65-7bc3-4760-90c6-0717a8a70502-kube-api-access-jhnh7\") pod \"maas-api-revoke-sub-b5cc3752-xt7z2\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.825028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.824982 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5cc3752-xt7z2\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.827729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.827707 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5cc3752-xt7z2\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.835449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.835420 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnh7\" (UniqueName: \"kubernetes.io/projected/df0a2d65-7bc3-4760-90c6-0717a8a70502-kube-api-access-jhnh7\") pod \"maas-api-revoke-sub-b5cc3752-xt7z2\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:22.886522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.886449 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:22.984329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:22.984299 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:23.032267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.032230 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-xks-ca\") pod \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " Nov 02 22:13:23.032464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.032419 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-openshift-ca\") pod \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " Nov 02 22:13:23.032523 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.032472 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-kube-api-access-2mcxs\" (UniqueName: \"kubernetes.io/projected/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-kube-api-access-2mcxs\") pod \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\" (UID: \"b1ee6f97-4e20-4fc7-a7fb-bc1288836488\") " Nov 02 22:13:23.033023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.032977 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-openshift-ca" pod "b1ee6f97-4e20-4fc7-a7fb-bc1288836488" (UID: "b1ee6f97-4e20-4fc7-a7fb-bc1288836488"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:23.034476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.034444 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-xks-ca" pod "b1ee6f97-4e20-4fc7-a7fb-bc1288836488" (UID: "b1ee6f97-4e20-4fc7-a7fb-bc1288836488"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:23.035290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.035265 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-kube-api-access-2mcxs" pod "b1ee6f97-4e20-4fc7-a7fb-bc1288836488" (UID: "b1ee6f97-4e20-4fc7-a7fb-bc1288836488"). InnerVolumeSpecName "kube-api-access-2mcxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:23.134333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.134150 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:23.134333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.134180 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:23.134333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.134192 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mcxs\" (UniqueName: \"kubernetes.io/projected/b1ee6f97-4e20-4fc7-a7fb-bc1288836488-kube-api-access-2mcxs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:23.276258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.276229 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2"] Nov 02 22:13:23.421696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.421295 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" event={"ID":"c35ad342-26b9-43a9-a327-02309df9f1dc","Type":"ContainerStarted","Data":"bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f"} Nov 02 22:13:23.435704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.428814 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" event={"ID":"df0a2d65-7bc3-4760-90c6-0717a8a70502","Type":"ContainerStarted","Data":"a64c4fe326132c354e1f4dd1f0e227a38b9b6dbf43b3e33a091e3f94866238b4"} Nov 02 22:13:23.435704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.432462 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" event={"ID":"b1ee6f97-4e20-4fc7-a7fb-bc1288836488","Type":"ContainerDied","Data":"b3721de76edda1bf0a9dcf9138925038c79b29335ac9a418e60317b6aa4bde06"} Nov 02 22:13:23.435704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.432514 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3721de76edda1bf0a9dcf9138925038c79b29335ac9a418e60317b6aa4bde06" Nov 02 22:13:23.435704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.432591 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2" Nov 02 22:13:23.456347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.455957 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podStartSLOduration=9.455927275 podStartE2EDuration="9.455927275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:23.445128767 +0000 UTC m=+931.383241986" watchObservedRunningTime="2026-11-02 22:13:23.455927275 +0000 UTC m=+931.394040494" Nov 02 22:13:23.474864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.474831 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc"] Nov 02 22:13:23.477057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.477028 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b1ee6f97-4e20-4fc7-a7fb-bc1288836488" containerName="revoke-keys" Nov 02 22:13:23.477146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.477059 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ee6f97-4e20-4fc7-a7fb-bc1288836488" containerName="revoke-keys" Nov 02 22:13:23.477290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.477275 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ee6f97-4e20-4fc7-a7fb-bc1288836488" containerName="revoke-keys" Nov 02 22:13:23.498589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.497626 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc"] Nov 02 22:13:23.498589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.497789 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.642835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.642800 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a28bbb0-q7btc\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.643104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.643082 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjw4\" (UniqueName: \"kubernetes.io/projected/79a3209f-5dac-4ae0-a0d6-1de62f2212be-kube-api-access-8rjw4\") pod \"maas-api-revoke-sub-8a28bbb0-q7btc\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.643214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.643193 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a28bbb0-q7btc\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.677046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.677019 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:23.678778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.678747 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:13:23.678902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.678793 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 22:13:23.744500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.744469 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjw4\" (UniqueName: \"kubernetes.io/projected/79a3209f-5dac-4ae0-a0d6-1de62f2212be-kube-api-access-8rjw4\") pod \"maas-api-revoke-sub-8a28bbb0-q7btc\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.744854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.744561 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a28bbb0-q7btc\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.744854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.744728 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a28bbb0-q7btc\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.745294 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.745274 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a28bbb0-q7btc\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.747053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.747032 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a28bbb0-q7btc\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.752043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.752016 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjw4\" (UniqueName: \"kubernetes.io/projected/79a3209f-5dac-4ae0-a0d6-1de62f2212be-kube-api-access-8rjw4\") pod \"maas-api-revoke-sub-8a28bbb0-q7btc\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:23.829127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:23.829092 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:24.242925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:24.242892 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc"] Nov 02 22:13:24.244276 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:24.244241 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a3209f_5dac_4ae0_a0d6_1de62f2212be.slice/crio-c47831c16c3a604b96da44e82a99cf2635f4bdf199210bd8d680e7b3d6a6cbb3 WatchSource:0}: Error finding container c47831c16c3a604b96da44e82a99cf2635f4bdf199210bd8d680e7b3d6a6cbb3: Status 404 returned error can't find the container with id c47831c16c3a604b96da44e82a99cf2635f4bdf199210bd8d680e7b3d6a6cbb3 Nov 02 22:13:24.464708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:24.464080 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" event={"ID":"79a3209f-5dac-4ae0-a0d6-1de62f2212be","Type":"ContainerStarted","Data":"c47831c16c3a604b96da44e82a99cf2635f4bdf199210bd8d680e7b3d6a6cbb3"} Nov 02 22:13:24.471400 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:24.471365 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" event={"ID":"df0a2d65-7bc3-4760-90c6-0717a8a70502","Type":"ContainerStarted","Data":"72447e44a5b22c318b7cdb72b37b8f7ce582e8f8b60a6075e2e71459f567fcc9"} Nov 02 22:13:24.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:24.522813 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" podStartSLOduration=2.522789112 podStartE2EDuration="2.522789112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:24.487074225 +0000 UTC m=+932.425187444" watchObservedRunningTime="2026-11-02 22:13:24.522789112 +0000 UTC m=+932.460902343" Nov 02 22:13:24.677241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:24.677205 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:13:24.677374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:24.677278 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 22:13:25.519736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:25.518212 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" event={"ID":"79a3209f-5dac-4ae0-a0d6-1de62f2212be","Type":"ContainerStarted","Data":"21ecb53249a0c18bffa983502d90ab4fe4cb3afc8ecf14ecfc7199774347209e"} Nov 02 22:13:25.526715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:25.524229 4309 generic.go:356] "Generic (PLEG): container finished" podID="df0a2d65-7bc3-4760-90c6-0717a8a70502" containerID="72447e44a5b22c318b7cdb72b37b8f7ce582e8f8b60a6075e2e71459f567fcc9" exitCode=0 Nov 02 22:13:25.526715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:25.525399 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" event={"ID":"df0a2d65-7bc3-4760-90c6-0717a8a70502","Type":"ContainerDied","Data":"72447e44a5b22c318b7cdb72b37b8f7ce582e8f8b60a6075e2e71459f567fcc9"} Nov 02 22:13:25.552567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:25.552029 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" podStartSLOduration=2.552010213 podStartE2EDuration="2.552010213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:25.534580623 +0000 UTC m=+933.472693875" watchObservedRunningTime="2026-11-02 22:13:25.552010213 +0000 UTC m=+933.490123435" Nov 02 22:13:25.677299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:25.677264 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:13:25.677445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:25.677318 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 22:13:26.064647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.064612 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn"] Nov 02 22:13:26.070146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.070114 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:26.078129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.078102 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn"] Nov 02 22:13:26.129044 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.127248 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ec62fcf4\"" Nov 02 22:13:26.129044 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.127542 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ec62fcf4-dockercfg-4jd97\"" Nov 02 22:13:26.227697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.227614 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd197b7-cb3d-4c50-95e1-f758cd47af00-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn\" (UID: \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\") " pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:26.227909 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.227727 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz64s\" (UniqueName: \"kubernetes.io/projected/7bd197b7-cb3d-4c50-95e1-f758cd47af00-kube-api-access-kz64s\") pod \"payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn\" (UID: \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\") " pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:26.341343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.341219 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd197b7-cb3d-4c50-95e1-f758cd47af00-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn\" (UID: \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\") " pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:26.342690 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.342646 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz64s\" (UniqueName: \"kubernetes.io/projected/7bd197b7-cb3d-4c50-95e1-f758cd47af00-kube-api-access-kz64s\") pod \"payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn\" (UID: \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\") " pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:26.343516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.342492 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd197b7-cb3d-4c50-95e1-f758cd47af00-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn\" (UID: \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\") " pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:26.355226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.355171 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz64s\" (UniqueName: \"kubernetes.io/projected/7bd197b7-cb3d-4c50-95e1-f758cd47af00-kube-api-access-kz64s\") pod \"payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn\" (UID: \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\") " pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:26.433270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.432974 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:26.584697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.581889 4309 generic.go:356] "Generic (PLEG): container finished" podID="79a3209f-5dac-4ae0-a0d6-1de62f2212be" containerID="21ecb53249a0c18bffa983502d90ab4fe4cb3afc8ecf14ecfc7199774347209e" exitCode=0 Nov 02 22:13:26.584697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.582396 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" event={"ID":"79a3209f-5dac-4ae0-a0d6-1de62f2212be","Type":"ContainerDied","Data":"21ecb53249a0c18bffa983502d90ab4fe4cb3afc8ecf14ecfc7199774347209e"} Nov 02 22:13:26.688750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.681653 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:13:26.688750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.681796 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 22:13:26.977226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:26.977185 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn"] Nov 02 22:13:27.034530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.034387 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:27.119707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.118537 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-xks-ca\") pod \"df0a2d65-7bc3-4760-90c6-0717a8a70502\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " Nov 02 22:13:27.119707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.118835 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-openshift-ca\") pod \"df0a2d65-7bc3-4760-90c6-0717a8a70502\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " Nov 02 22:13:27.119707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.119044 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0a2d65-7bc3-4760-90c6-0717a8a70502-kube-api-access-jhnh7\" (UniqueName: \"kubernetes.io/projected/df0a2d65-7bc3-4760-90c6-0717a8a70502-kube-api-access-jhnh7\") pod \"df0a2d65-7bc3-4760-90c6-0717a8a70502\" (UID: \"df0a2d65-7bc3-4760-90c6-0717a8a70502\") " Nov 02 22:13:27.128510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.128476 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-openshift-ca" pod "df0a2d65-7bc3-4760-90c6-0717a8a70502" (UID: "df0a2d65-7bc3-4760-90c6-0717a8a70502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:27.139730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.138244 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0a2d65-7bc3-4760-90c6-0717a8a70502-kube-api-access-jhnh7" pod "df0a2d65-7bc3-4760-90c6-0717a8a70502" (UID: "df0a2d65-7bc3-4760-90c6-0717a8a70502"). InnerVolumeSpecName "kube-api-access-jhnh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:27.139730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.139069 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-xks-ca" pod "df0a2d65-7bc3-4760-90c6-0717a8a70502" (UID: "df0a2d65-7bc3-4760-90c6-0717a8a70502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:27.221151 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.221069 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnh7\" (UniqueName: \"kubernetes.io/projected/df0a2d65-7bc3-4760-90c6-0717a8a70502-kube-api-access-jhnh7\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:27.221151 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.221103 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:27.221151 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.221115 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0a2d65-7bc3-4760-90c6-0717a8a70502-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:27.626611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.626578 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9"] Nov 02 22:13:27.627958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.627930 4309 cpu_manager.go:414] "containerMap: removing container" podUID="df0a2d65-7bc3-4760-90c6-0717a8a70502" containerName="revoke-keys" Nov 02 22:13:27.627958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.627957 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0a2d65-7bc3-4760-90c6-0717a8a70502" containerName="revoke-keys" Nov 02 22:13:27.628235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.628216 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0a2d65-7bc3-4760-90c6-0717a8a70502" containerName="revoke-keys" Nov 02 22:13:27.633494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.633473 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:27.633631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.633508 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" event={"ID":"7bd197b7-cb3d-4c50-95e1-f758cd47af00","Type":"ContainerStarted","Data":"f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6"} Nov 02 22:13:27.633631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.633544 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" event={"ID":"7bd197b7-cb3d-4c50-95e1-f758cd47af00","Type":"ContainerStarted","Data":"6f3661526223ce484a03210c3024eb1975e7ccdfe40f365b720acf3bdbe65cbe"} Nov 02 22:13:27.633838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.633631 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:27.635436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.635408 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" Nov 02 22:13:27.635635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.635606 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2" event={"ID":"df0a2d65-7bc3-4760-90c6-0717a8a70502","Type":"ContainerDied","Data":"a64c4fe326132c354e1f4dd1f0e227a38b9b6dbf43b3e33a091e3f94866238b4"} Nov 02 22:13:27.635855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.635838 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a64c4fe326132c354e1f4dd1f0e227a38b9b6dbf43b3e33a091e3f94866238b4" Nov 02 22:13:27.639032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.638932 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9"] Nov 02 22:13:27.654206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.652783 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" podStartSLOduration=1.652768736 podStartE2EDuration="1.652768736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:27.65203246 +0000 UTC m=+935.590145680" watchObservedRunningTime="2026-11-02 22:13:27.652768736 +0000 UTC m=+935.590881954" Nov 02 22:13:27.679002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.678961 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:13:27.679148 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.679023 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 22:13:27.732954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.732906 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac617154-9mcj9\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:27.733119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.733022 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhs8f\" (UniqueName: \"kubernetes.io/projected/901b6eb1-9e1b-4442-bdc5-e5cda2707940-kube-api-access-mhs8f\") pod \"maas-api-revoke-sub-ac617154-9mcj9\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:27.733212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.733191 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac617154-9mcj9\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:27.869862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.867490 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhs8f\" (UniqueName: \"kubernetes.io/projected/901b6eb1-9e1b-4442-bdc5-e5cda2707940-kube-api-access-mhs8f\") pod \"maas-api-revoke-sub-ac617154-9mcj9\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:27.869862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.867701 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac617154-9mcj9\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:27.869862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.867983 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac617154-9mcj9\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:27.869862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.868954 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac617154-9mcj9\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:27.881119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.881011 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhs8f\" (UniqueName: \"kubernetes.io/projected/901b6eb1-9e1b-4442-bdc5-e5cda2707940-kube-api-access-mhs8f\") pod \"maas-api-revoke-sub-ac617154-9mcj9\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:27.970241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:27.970213 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac617154-9mcj9\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:28.029951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.029922 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:28.173453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.173378 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a3209f-5dac-4ae0-a0d6-1de62f2212be-kube-api-access-8rjw4\" (UniqueName: \"kubernetes.io/projected/79a3209f-5dac-4ae0-a0d6-1de62f2212be-kube-api-access-8rjw4\") pod \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " Nov 02 22:13:28.173586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.173510 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-openshift-ca\") pod \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " Nov 02 22:13:28.173708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.173660 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-xks-ca\") pod \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\" (UID: \"79a3209f-5dac-4ae0-a0d6-1de62f2212be\") " Nov 02 22:13:28.173924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.173901 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-openshift-ca" pod "79a3209f-5dac-4ae0-a0d6-1de62f2212be" (UID: "79a3209f-5dac-4ae0-a0d6-1de62f2212be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:28.174341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.174317 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:28.175387 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.175365 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a3209f-5dac-4ae0-a0d6-1de62f2212be-kube-api-access-8rjw4" pod "79a3209f-5dac-4ae0-a0d6-1de62f2212be" (UID: "79a3209f-5dac-4ae0-a0d6-1de62f2212be"). InnerVolumeSpecName "kube-api-access-8rjw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:28.175620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.175595 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-xks-ca" pod "79a3209f-5dac-4ae0-a0d6-1de62f2212be" (UID: "79a3209f-5dac-4ae0-a0d6-1de62f2212be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:28.260809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.260781 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:28.275729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.275659 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rjw4\" (UniqueName: \"kubernetes.io/projected/79a3209f-5dac-4ae0-a0d6-1de62f2212be-kube-api-access-8rjw4\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:28.275729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.275725 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79a3209f-5dac-4ae0-a0d6-1de62f2212be-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:28.628651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.628608 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9"] Nov 02 22:13:28.660014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.659983 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" event={"ID":"901b6eb1-9e1b-4442-bdc5-e5cda2707940","Type":"ContainerStarted","Data":"16b759468a73bd864776f78ed6f58b37a700e4f154d710488bcd913eaa1c8272"} Nov 02 22:13:28.662421 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.662387 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" event={"ID":"79a3209f-5dac-4ae0-a0d6-1de62f2212be","Type":"ContainerDied","Data":"c47831c16c3a604b96da44e82a99cf2635f4bdf199210bd8d680e7b3d6a6cbb3"} Nov 02 22:13:28.662583 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.662442 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c47831c16c3a604b96da44e82a99cf2635f4bdf199210bd8d680e7b3d6a6cbb3" Nov 02 22:13:28.662702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.662666 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc" Nov 02 22:13:28.677840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.677815 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:13:28.677979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:28.677869 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 22:13:29.672089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.672052 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" event={"ID":"901b6eb1-9e1b-4442-bdc5-e5cda2707940","Type":"ContainerStarted","Data":"c0ea220d275d73b1d658e451ca8fa0a3fa1f4aeeb7827e251d34ed398988000f"} Nov 02 22:13:29.677618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.677590 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 02 22:13:29.677763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.677636 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 02 22:13:29.695275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.695215 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" podStartSLOduration=2.6951963450000003 podStartE2EDuration="2.695196345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:29.688732501 +0000 UTC m=+937.626845720" watchObservedRunningTime="2026-11-02 22:13:29.695196345 +0000 UTC m=+937.633309564" Nov 02 22:13:29.879502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.879458 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm"] Nov 02 22:13:29.881234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.880715 4309 cpu_manager.go:414] "containerMap: removing container" podUID="79a3209f-5dac-4ae0-a0d6-1de62f2212be" containerName="revoke-keys" Nov 02 22:13:29.881234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.880739 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a3209f-5dac-4ae0-a0d6-1de62f2212be" containerName="revoke-keys" Nov 02 22:13:29.881234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.880930 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a3209f-5dac-4ae0-a0d6-1de62f2212be" containerName="revoke-keys" Nov 02 22:13:29.887729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.885891 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:29.890213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.890105 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm"] Nov 02 22:13:29.933789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.929608 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f934efa9-7glmm\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:29.933789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.929905 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f78pb\" (UniqueName: \"kubernetes.io/projected/58553e2b-8414-4b6f-8682-0547655fa062-kube-api-access-f78pb\") pod \"maas-api-revoke-sub-f934efa9-7glmm\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:29.933789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:29.929964 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f934efa9-7glmm\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:30.031439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.031398 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f78pb\" (UniqueName: \"kubernetes.io/projected/58553e2b-8414-4b6f-8682-0547655fa062-kube-api-access-f78pb\") pod \"maas-api-revoke-sub-f934efa9-7glmm\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:30.031439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.031458 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f934efa9-7glmm\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:30.031762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.031671 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f934efa9-7glmm\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:30.032349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.032324 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f934efa9-7glmm\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:30.034941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.034915 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f934efa9-7glmm\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:30.061160 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.061124 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f78pb\" (UniqueName: \"kubernetes.io/projected/58553e2b-8414-4b6f-8682-0547655fa062-kube-api-access-f78pb\") pod \"maas-api-revoke-sub-f934efa9-7glmm\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:30.233972 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.233890 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:30.538414 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:30.538382 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58553e2b_8414_4b6f_8682_0547655fa062.slice/crio-0ddc8ea415aeea042c335219200dd866a2c221d7d862b4b2b20aa4acacbc5141 WatchSource:0}: Error finding container 0ddc8ea415aeea042c335219200dd866a2c221d7d862b4b2b20aa4acacbc5141: Status 404 returned error can't find the container with id 0ddc8ea415aeea042c335219200dd866a2c221d7d862b4b2b20aa4acacbc5141 Nov 02 22:13:30.539981 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.539956 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm"] Nov 02 22:13:30.676970 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.676622 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:30.681533 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.681502 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:30.683463 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.683435 4309 generic.go:356] "Generic (PLEG): container finished" podID="901b6eb1-9e1b-4442-bdc5-e5cda2707940" containerID="c0ea220d275d73b1d658e451ca8fa0a3fa1f4aeeb7827e251d34ed398988000f" exitCode=0 Nov 02 22:13:30.683596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.683523 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" event={"ID":"901b6eb1-9e1b-4442-bdc5-e5cda2707940","Type":"ContainerDied","Data":"c0ea220d275d73b1d658e451ca8fa0a3fa1f4aeeb7827e251d34ed398988000f"} Nov 02 22:13:30.685093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:30.685071 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" event={"ID":"58553e2b-8414-4b6f-8682-0547655fa062","Type":"ContainerStarted","Data":"0ddc8ea415aeea042c335219200dd866a2c221d7d862b4b2b20aa4acacbc5141"} Nov 02 22:13:31.699075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:31.699022 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" event={"ID":"58553e2b-8414-4b6f-8682-0547655fa062","Type":"ContainerStarted","Data":"6a8f4fd64ff5de8aa02177c3cb0a1373291492e0f7ffc1fdd7f8c519f5be5dfd"} Nov 02 22:13:31.701120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:31.701093 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:13:32.022590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.022563 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:32.081841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.081786 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-openshift-ca\") pod \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " Nov 02 22:13:32.082081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.082061 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901b6eb1-9e1b-4442-bdc5-e5cda2707940-kube-api-access-mhs8f\" (UniqueName: \"kubernetes.io/projected/901b6eb1-9e1b-4442-bdc5-e5cda2707940-kube-api-access-mhs8f\") pod \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " Nov 02 22:13:32.082213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.082190 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-xks-ca\") pod \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\" (UID: \"901b6eb1-9e1b-4442-bdc5-e5cda2707940\") " Nov 02 22:13:32.083720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.083649 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-openshift-ca" pod "901b6eb1-9e1b-4442-bdc5-e5cda2707940" (UID: "901b6eb1-9e1b-4442-bdc5-e5cda2707940"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:32.085616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.085590 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-xks-ca" pod "901b6eb1-9e1b-4442-bdc5-e5cda2707940" (UID: "901b6eb1-9e1b-4442-bdc5-e5cda2707940"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:32.096800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.093165 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901b6eb1-9e1b-4442-bdc5-e5cda2707940-kube-api-access-mhs8f" pod "901b6eb1-9e1b-4442-bdc5-e5cda2707940" (UID: "901b6eb1-9e1b-4442-bdc5-e5cda2707940"). InnerVolumeSpecName "kube-api-access-mhs8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:32.184769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.184733 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhs8f\" (UniqueName: \"kubernetes.io/projected/901b6eb1-9e1b-4442-bdc5-e5cda2707940-kube-api-access-mhs8f\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:32.184769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.184768 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:32.184965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.184784 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6eb1-9e1b-4442-bdc5-e5cda2707940-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:32.464384 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:32.464344 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:32.739650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.739534 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" Nov 02 22:13:32.739650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.739549 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9" event={"ID":"901b6eb1-9e1b-4442-bdc5-e5cda2707940","Type":"ContainerDied","Data":"16b759468a73bd864776f78ed6f58b37a700e4f154d710488bcd913eaa1c8272"} Nov 02 22:13:32.739650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.739598 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16b759468a73bd864776f78ed6f58b37a700e4f154d710488bcd913eaa1c8272" Nov 02 22:13:32.741853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.741821 4309 generic.go:356] "Generic (PLEG): container finished" podID="58553e2b-8414-4b6f-8682-0547655fa062" containerID="6a8f4fd64ff5de8aa02177c3cb0a1373291492e0f7ffc1fdd7f8c519f5be5dfd" exitCode=0 Nov 02 22:13:32.742021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.741912 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" event={"ID":"58553e2b-8414-4b6f-8682-0547655fa062","Type":"ContainerDied","Data":"6a8f4fd64ff5de8aa02177c3cb0a1373291492e0f7ffc1fdd7f8c519f5be5dfd"} Nov 02 22:13:32.752822 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.752790 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz"] Nov 02 22:13:32.753629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.753607 4309 cpu_manager.go:414] "containerMap: removing container" podUID="901b6eb1-9e1b-4442-bdc5-e5cda2707940" containerName="revoke-keys" Nov 02 22:13:32.753629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.753628 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901b6eb1-9e1b-4442-bdc5-e5cda2707940" containerName="revoke-keys" Nov 02 22:13:32.753843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.753828 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="901b6eb1-9e1b-4442-bdc5-e5cda2707940" containerName="revoke-keys" Nov 02 22:13:32.758380 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.758355 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:32.765878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.765298 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz"] Nov 02 22:13:32.902759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.899142 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f6704a-qrqrz\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:32.902759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.899195 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f6704a-qrqrz\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:32.902759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.899297 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2knl\" (UniqueName: \"kubernetes.io/projected/f4e154eb-fb69-4a0a-b311-f4db51e64c21-kube-api-access-n2knl\") pod \"maas-api-revoke-sub-10f6704a-qrqrz\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:32.956720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:32.956687 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:33.006864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.006774 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f6704a-qrqrz\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:33.006864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.006853 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f6704a-qrqrz\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:33.007111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.006969 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2knl\" (UniqueName: \"kubernetes.io/projected/f4e154eb-fb69-4a0a-b311-f4db51e64c21-kube-api-access-n2knl\") pod \"maas-api-revoke-sub-10f6704a-qrqrz\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:33.007993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.007948 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f6704a-qrqrz\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:33.010141 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.009831 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f6704a-qrqrz\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:33.015497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.015474 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2knl\" (UniqueName: \"kubernetes.io/projected/f4e154eb-fb69-4a0a-b311-f4db51e64c21-kube-api-access-n2knl\") pod \"maas-api-revoke-sub-10f6704a-qrqrz\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:33.079596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.079559 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:33.107991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.107961 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-xks-ca\") pod \"58553e2b-8414-4b6f-8682-0547655fa062\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " Nov 02 22:13:33.108219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.108201 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58553e2b-8414-4b6f-8682-0547655fa062-kube-api-access-f78pb\" (UniqueName: \"kubernetes.io/projected/58553e2b-8414-4b6f-8682-0547655fa062-kube-api-access-f78pb\") pod \"58553e2b-8414-4b6f-8682-0547655fa062\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " Nov 02 22:13:33.108262 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.108250 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-openshift-ca\") pod \"58553e2b-8414-4b6f-8682-0547655fa062\" (UID: \"58553e2b-8414-4b6f-8682-0547655fa062\") " Nov 02 22:13:33.108921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.108886 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-openshift-ca" pod "58553e2b-8414-4b6f-8682-0547655fa062" (UID: "58553e2b-8414-4b6f-8682-0547655fa062"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:33.109198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.109171 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:33.110738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.110712 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58553e2b-8414-4b6f-8682-0547655fa062-kube-api-access-f78pb" pod "58553e2b-8414-4b6f-8682-0547655fa062" (UID: "58553e2b-8414-4b6f-8682-0547655fa062"). InnerVolumeSpecName "kube-api-access-f78pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:33.110831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.110711 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-xks-ca" pod "58553e2b-8414-4b6f-8682-0547655fa062" (UID: "58553e2b-8414-4b6f-8682-0547655fa062"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:33.183370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.183340 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk"] Nov 02 22:13:33.184140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.184124 4309 cpu_manager.go:414] "containerMap: removing container" podUID="58553e2b-8414-4b6f-8682-0547655fa062" containerName="revoke-keys" Nov 02 22:13:33.184200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.184142 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58553e2b-8414-4b6f-8682-0547655fa062" containerName="revoke-keys" Nov 02 22:13:33.184264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.184254 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="58553e2b-8414-4b6f-8682-0547655fa062" containerName="revoke-keys" Nov 02 22:13:33.189856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.189834 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.202216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.202188 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk"] Nov 02 22:13:33.212889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.211068 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f78pb\" (UniqueName: \"kubernetes.io/projected/58553e2b-8414-4b6f-8682-0547655fa062-kube-api-access-f78pb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:33.212889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.211099 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58553e2b-8414-4b6f-8682-0547655fa062-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:33.318411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.318378 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2tt\" (UniqueName: \"kubernetes.io/projected/405643ba-bf63-46f9-b0c4-d57622186f6a-kube-api-access-pv2tt\") pod \"maas-api-revoke-sub-9ce66a83-qhrfk\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.318612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.318542 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce66a83-qhrfk\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.318800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.318780 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce66a83-qhrfk\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.377060 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:33.377016 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e154eb_fb69_4a0a_b311_f4db51e64c21.slice/crio-fd4a71db076019a03eb876c9aa25c84a0db20471a1f783c3d0dc187f45f84532 WatchSource:0}: Error finding container fd4a71db076019a03eb876c9aa25c84a0db20471a1f783c3d0dc187f45f84532: Status 404 returned error can't find the container with id fd4a71db076019a03eb876c9aa25c84a0db20471a1f783c3d0dc187f45f84532 Nov 02 22:13:33.381154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.381126 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz"] Nov 02 22:13:33.420432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.420400 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2tt\" (UniqueName: \"kubernetes.io/projected/405643ba-bf63-46f9-b0c4-d57622186f6a-kube-api-access-pv2tt\") pod \"maas-api-revoke-sub-9ce66a83-qhrfk\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.420549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.420516 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce66a83-qhrfk\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.420654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.420637 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce66a83-qhrfk\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.421617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.421576 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce66a83-qhrfk\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.423359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.423339 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce66a83-qhrfk\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.429242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.429202 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2tt\" (UniqueName: \"kubernetes.io/projected/405643ba-bf63-46f9-b0c4-d57622186f6a-kube-api-access-pv2tt\") pod \"maas-api-revoke-sub-9ce66a83-qhrfk\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.520621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.520583 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:33.653616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.653425 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv"] Nov 02 22:13:33.666180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.665748 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv"] Nov 02 22:13:33.666180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.665862 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.752492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.752462 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" Nov 02 22:13:33.752492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.752478 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm" event={"ID":"58553e2b-8414-4b6f-8682-0547655fa062","Type":"ContainerDied","Data":"0ddc8ea415aeea042c335219200dd866a2c221d7d862b4b2b20aa4acacbc5141"} Nov 02 22:13:33.752980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.752530 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ddc8ea415aeea042c335219200dd866a2c221d7d862b4b2b20aa4acacbc5141" Nov 02 22:13:33.755047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.755019 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" event={"ID":"f4e154eb-fb69-4a0a-b311-f4db51e64c21","Type":"ContainerStarted","Data":"fd4a71db076019a03eb876c9aa25c84a0db20471a1f783c3d0dc187f45f84532"} Nov 02 22:13:33.817585 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:33.817559 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405643ba_bf63_46f9_b0c4_d57622186f6a.slice/crio-ecf24adcbadec6d4280754957be9dce3b87ba0342495c6d6d02f10001b026584 WatchSource:0}: Error finding container ecf24adcbadec6d4280754957be9dce3b87ba0342495c6d6d02f10001b026584: Status 404 returned error can't find the container with id ecf24adcbadec6d4280754957be9dce3b87ba0342495c6d6d02f10001b026584 Nov 02 22:13:33.820985 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.820959 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk"] Nov 02 22:13:33.834725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.834658 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/e4fd685b-1edf-42fb-b508-bfb91a72d32c-kube-api-access-kx698\") pod \"maas-api-revoke-sub-22358063-f26pv\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.834844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.834772 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22358063-f26pv\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.834844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.834820 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22358063-f26pv\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.936159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.936124 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/e4fd685b-1edf-42fb-b508-bfb91a72d32c-kube-api-access-kx698\") pod \"maas-api-revoke-sub-22358063-f26pv\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.936297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.936206 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22358063-f26pv\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.936297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.936241 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22358063-f26pv\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.936852 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.936832 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22358063-f26pv\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.938471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.938453 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22358063-f26pv\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.945055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.945034 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/e4fd685b-1edf-42fb-b508-bfb91a72d32c-kube-api-access-kx698\") pod \"maas-api-revoke-sub-22358063-f26pv\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:33.990417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:33.990086 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:34.314834 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:34.314803 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4fd685b_1edf_42fb_b508_bfb91a72d32c.slice/crio-d504909740a752b7cca958ce858c83ae6b00c90170e56656a1b7d47cddedb2e2 WatchSource:0}: Error finding container d504909740a752b7cca958ce858c83ae6b00c90170e56656a1b7d47cddedb2e2: Status 404 returned error can't find the container with id d504909740a752b7cca958ce858c83ae6b00c90170e56656a1b7d47cddedb2e2 Nov 02 22:13:34.316344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:34.316321 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv"] Nov 02 22:13:34.774166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:34.774059 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" event={"ID":"405643ba-bf63-46f9-b0c4-d57622186f6a","Type":"ContainerStarted","Data":"0abc7e1ff1eb3430dcecfe01169bab2b74469e159160fca06ab0fa2acbb34083"} Nov 02 22:13:34.774166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:34.774127 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" event={"ID":"405643ba-bf63-46f9-b0c4-d57622186f6a","Type":"ContainerStarted","Data":"ecf24adcbadec6d4280754957be9dce3b87ba0342495c6d6d02f10001b026584"} Nov 02 22:13:34.778494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:34.776464 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" event={"ID":"e4fd685b-1edf-42fb-b508-bfb91a72d32c","Type":"ContainerStarted","Data":"d504909740a752b7cca958ce858c83ae6b00c90170e56656a1b7d47cddedb2e2"} Nov 02 22:13:34.801992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:34.801955 4309 generic.go:356] "Generic (PLEG): container finished" podID="f4e154eb-fb69-4a0a-b311-f4db51e64c21" containerID="294338792e085541a0d107b39f091f69ed630200cb0ee5c1f007fb4dbcadb14e" exitCode=0 Nov 02 22:13:34.802185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:34.802055 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" event={"ID":"f4e154eb-fb69-4a0a-b311-f4db51e64c21","Type":"ContainerDied","Data":"294338792e085541a0d107b39f091f69ed630200cb0ee5c1f007fb4dbcadb14e"} Nov 02 22:13:34.803405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:34.803351 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" podStartSLOduration=1.803333506 podStartE2EDuration="1.803333506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:34.802375634 +0000 UTC m=+942.740488851" watchObservedRunningTime="2026-11-02 22:13:34.803333506 +0000 UTC m=+942.741446725" Nov 02 22:13:35.817273 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:35.817239 4309 generic.go:356] "Generic (PLEG): container finished" podID="e4fd685b-1edf-42fb-b508-bfb91a72d32c" containerID="867f0012c557c399e4e5d1b46da705399efacdc410bf935c824f5d41b3cd3ea3" exitCode=0 Nov 02 22:13:35.817765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:35.817320 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" event={"ID":"e4fd685b-1edf-42fb-b508-bfb91a72d32c","Type":"ContainerDied","Data":"867f0012c557c399e4e5d1b46da705399efacdc410bf935c824f5d41b3cd3ea3"} Nov 02 22:13:35.820358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:35.820334 4309 generic.go:356] "Generic (PLEG): container finished" podID="405643ba-bf63-46f9-b0c4-d57622186f6a" containerID="0abc7e1ff1eb3430dcecfe01169bab2b74469e159160fca06ab0fa2acbb34083" exitCode=0 Nov 02 22:13:35.820480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:35.820421 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" event={"ID":"405643ba-bf63-46f9-b0c4-d57622186f6a","Type":"ContainerDied","Data":"0abc7e1ff1eb3430dcecfe01169bab2b74469e159160fca06ab0fa2acbb34083"} Nov 02 22:13:35.914978 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:35.914933 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:36.015272 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.015237 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:36.189994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.189888 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-xks-ca\") pod \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " Nov 02 22:13:36.189994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.189952 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-openshift-ca\") pod \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " Nov 02 22:13:36.190238 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.190100 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e154eb-fb69-4a0a-b311-f4db51e64c21-kube-api-access-n2knl\" (UniqueName: \"kubernetes.io/projected/f4e154eb-fb69-4a0a-b311-f4db51e64c21-kube-api-access-n2knl\") pod \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\" (UID: \"f4e154eb-fb69-4a0a-b311-f4db51e64c21\") " Nov 02 22:13:36.190897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.190830 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-openshift-ca" pod "f4e154eb-fb69-4a0a-b311-f4db51e64c21" (UID: "f4e154eb-fb69-4a0a-b311-f4db51e64c21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:36.191099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.191066 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:36.192339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.192312 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-xks-ca" pod "f4e154eb-fb69-4a0a-b311-f4db51e64c21" (UID: "f4e154eb-fb69-4a0a-b311-f4db51e64c21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:36.192515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.192494 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e154eb-fb69-4a0a-b311-f4db51e64c21-kube-api-access-n2knl" pod "f4e154eb-fb69-4a0a-b311-f4db51e64c21" (UID: "f4e154eb-fb69-4a0a-b311-f4db51e64c21"). InnerVolumeSpecName "kube-api-access-n2knl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:36.292241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.292207 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4e154eb-fb69-4a0a-b311-f4db51e64c21-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:36.292241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.292237 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2knl\" (UniqueName: \"kubernetes.io/projected/f4e154eb-fb69-4a0a-b311-f4db51e64c21-kube-api-access-n2knl\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:36.834762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.834725 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" event={"ID":"f4e154eb-fb69-4a0a-b311-f4db51e64c21","Type":"ContainerDied","Data":"fd4a71db076019a03eb876c9aa25c84a0db20471a1f783c3d0dc187f45f84532"} Nov 02 22:13:36.834762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.834755 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz" Nov 02 22:13:36.835293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:36.834786 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd4a71db076019a03eb876c9aa25c84a0db20471a1f783c3d0dc187f45f84532" Nov 02 22:13:38.252525 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.252487 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.372711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.347171 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.372711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.347548 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:38.501741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.478066 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:38.502092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.502050 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.578756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.574918 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.609022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.606153 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" podUID="96e4208b-1bbe-4a26-a5c0-77783d8239b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded" Nov 02 22:13:38.609614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.609433 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.650845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.639001 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:13:38.650845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.648889 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.651979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.651949 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-xks-ca\") pod \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " Nov 02 22:13:38.652209 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.652186 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-openshift-ca\") pod \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " Nov 02 22:13:38.673701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.652594 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4fd685b-1edf-42fb-b508-bfb91a72d32c-kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/e4fd685b-1edf-42fb-b508-bfb91a72d32c-kube-api-access-kx698\") pod \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\" (UID: \"e4fd685b-1edf-42fb-b508-bfb91a72d32c\") " Nov 02 22:13:38.678193 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.678162 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" event={"ID":"e4fd685b-1edf-42fb-b508-bfb91a72d32c","Type":"ContainerDied","Data":"d504909740a752b7cca958ce858c83ae6b00c90170e56656a1b7d47cddedb2e2"} Nov 02 22:13:38.683779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.683750 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d504909740a752b7cca958ce858c83ae6b00c90170e56656a1b7d47cddedb2e2" Nov 02 22:13:38.742084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.720659 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.862704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.773941 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:13:38.911727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.790627 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:38.911727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.791863 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-openshift-ca" pod "e4fd685b-1edf-42fb-b508-bfb91a72d32c" (UID: "e4fd685b-1edf-42fb-b508-bfb91a72d32c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:38.911727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.844017 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:13:39.008701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.946460 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:39.096761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:38.985146 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fd685b-1edf-42fb-b508-bfb91a72d32c-kube-api-access-kx698" pod "e4fd685b-1edf-42fb-b508-bfb91a72d32c" (UID: "e4fd685b-1edf-42fb-b508-bfb91a72d32c"). InnerVolumeSpecName "kube-api-access-kx698". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:39.096761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.028816 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-xks-ca" pod "e4fd685b-1edf-42fb-b508-bfb91a72d32c" (UID: "e4fd685b-1edf-42fb-b508-bfb91a72d32c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:39.104712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.102007 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:39.104712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.102072 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:39.261702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.258894 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:13:39.261702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.259585 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:13:39.382769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.379131 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:39.413013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.406980 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:13:39.416919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.413995 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:39.416919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.415984 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:13:39.531205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.531151 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:39.638763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.569496 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:39.638763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.636617 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:39.645706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.639412 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:13:39.672593 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.646283 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:13:39.725701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.706334 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:13:39.800731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.798852 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:13:39.935763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.913017 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:13:39.986696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:39.980050 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:40.063795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.025935 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:13:40.063795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.040737 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4fd685b-1edf-42fb-b508-bfb91a72d32c-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:40.063795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.041856 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx698\" (UniqueName: \"kubernetes.io/projected/e4fd685b-1edf-42fb-b508-bfb91a72d32c-kube-api-access-kx698\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:40.089842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.089799 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:40.208752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.194980 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 02 22:13:40.208752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.195697 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:13:40.208752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.204763 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:40.307742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.251879 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:40.307742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.263025 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:40.307742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.268577 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:13:40.336702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.323987 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:13:40.336702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.328484 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:13:40.362446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.362411 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:13:40.362831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.362809 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:13:40.440516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.440484 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:40.441247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.441214 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:40.616703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.590093 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:13:40.756712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.733176 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:41.013760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.804320 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:13:41.013760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:40.921409 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:13:41.018455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.018419 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-openshift-ca\") pod \"405643ba-bf63-46f9-b0c4-d57622186f6a\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " Nov 02 22:13:41.033190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.033154 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-xks-ca\") pod \"405643ba-bf63-46f9-b0c4-d57622186f6a\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " Nov 02 22:13:41.056701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.030071 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-openshift-ca" pod "405643ba-bf63-46f9-b0c4-d57622186f6a" (UID: "405643ba-bf63-46f9-b0c4-d57622186f6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:41.098485 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.098449 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv" Nov 02 22:13:41.176621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.143992 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405643ba-bf63-46f9-b0c4-d57622186f6a-kube-api-access-pv2tt\" (UniqueName: \"kubernetes.io/projected/405643ba-bf63-46f9-b0c4-d57622186f6a-kube-api-access-pv2tt\") pod \"405643ba-bf63-46f9-b0c4-d57622186f6a\" (UID: \"405643ba-bf63-46f9-b0c4-d57622186f6a\") " Nov 02 22:13:41.203333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.102948 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" Nov 02 22:13:41.225704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.214359 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:41.263418 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.263366 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:41.292615 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.292416 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:41.321347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.321297 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405643ba-bf63-46f9-b0c4-d57622186f6a-kube-api-access-pv2tt" pod "405643ba-bf63-46f9-b0c4-d57622186f6a" (UID: "405643ba-bf63-46f9-b0c4-d57622186f6a"). InnerVolumeSpecName "kube-api-access-pv2tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:41.348700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.325554 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv2tt\" (UniqueName: \"kubernetes.io/projected/405643ba-bf63-46f9-b0c4-d57622186f6a-kube-api-access-pv2tt\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:41.423049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.423005 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:13:41.451926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.451860 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:13:41.603757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.268841 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-xks-ca" pod "405643ba-bf63-46f9-b0c4-d57622186f6a" (UID: "405643ba-bf63-46f9-b0c4-d57622186f6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:41.663812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.597777 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:41.687823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.326840 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:42.120328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.675746 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:42.125710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.761547 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405643ba-bf63-46f9-b0c4-d57622186f6a-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:42.129706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.906219 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:13:42.286785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.907461 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:42.293705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.934952 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:13:42.294211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.990120 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:42.294501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:41.990154 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:13:42.298800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:42.075693 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" podUID="3768a351-a0c8-4472-8769-d39331d91f5a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:42.299313 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:42.106083 4309 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.739s" Nov 02 22:13:42.340717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:42.265007 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:42.340717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:42.293968 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:42.464839 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:42.464790 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:42.514851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:42.514803 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:42.688932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:42.688887 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:42.986798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:42.985524 4309 patch_prober.go:29] interesting pod/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:42.986798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:42.985609 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:43.010701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:42.987232 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Nov 02 22:13:43.128906 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.128843 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:43.130589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.130546 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:43.131800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.131772 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:13:43.259645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.259551 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:43.259905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.259872 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:43.291879 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:43.291841 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:13:43.350033 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:43.335406 4309 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.03s" Nov 02 22:13:43.350853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.350806 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk" event={"ID":"405643ba-bf63-46f9-b0c4-d57622186f6a","Type":"ContainerDied","Data":"ecf24adcbadec6d4280754957be9dce3b87ba0342495c6d6d02f10001b026584"} Nov 02 22:13:43.351046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.351030 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecf24adcbadec6d4280754957be9dce3b87ba0342495c6d6d02f10001b026584" Nov 02 22:13:43.351243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.351232 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:13:43.461756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.351526 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:13:43.807007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.419053 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:43.807372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.419092 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:43.807625 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.419113 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:13:43.807906 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.419132 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:13:43.808131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.419151 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:13:43.942764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:43.938700 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp"] Nov 02 22:13:44.322707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.304355 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f4e154eb-fb69-4a0a-b311-f4db51e64c21" containerName="revoke-keys" Nov 02 22:13:44.322707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.304388 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e154eb-fb69-4a0a-b311-f4db51e64c21" containerName="revoke-keys" Nov 02 22:13:44.322707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.304424 4309 cpu_manager.go:414] "containerMap: removing container" podUID="405643ba-bf63-46f9-b0c4-d57622186f6a" containerName="revoke-keys" Nov 02 22:13:44.322707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.304434 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405643ba-bf63-46f9-b0c4-d57622186f6a" containerName="revoke-keys" Nov 02 22:13:44.322707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.304475 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e4fd685b-1edf-42fb-b508-bfb91a72d32c" containerName="revoke-keys" Nov 02 22:13:44.322707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.304484 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4fd685b-1edf-42fb-b508-bfb91a72d32c" containerName="revoke-keys" Nov 02 22:13:44.352695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.284118 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:13:44.413870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.399363 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:44.417700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.414259 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e154eb-fb69-4a0a-b311-f4db51e64c21" containerName="revoke-keys" Nov 02 22:13:44.765760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.758361 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="405643ba-bf63-46f9-b0c4-d57622186f6a" containerName="revoke-keys" Nov 02 22:13:44.780706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.767054 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4fd685b-1edf-42fb-b508-bfb91a72d32c" containerName="revoke-keys" Nov 02 22:13:44.862783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.641202 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.692802 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" podUID="d2b2a428-093b-4d9c-8b88-ad2b24b447c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.557986 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": context deadline exceeded" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.558170 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.185585 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.111895 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.049198 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.185326 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.185074 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.384655 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.049410 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:44.185801 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:45.066758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.046963 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:13:45.111706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.096168 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:13:45.230701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.123496 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:13:45.230701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.125058 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:13:45.230701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.169140 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:13:45.230701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.212305 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" podUID="3768a351-a0c8-4472-8769-d39331d91f5a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:45.230701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.221731 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:45.341171 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.336900 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:45.508155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.505202 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0f9863c-9kglp\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:45.521902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.521856 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnjm\" (UniqueName: \"kubernetes.io/projected/43039834-e511-4d83-b65d-51e6233b1b7a-kube-api-access-6gnjm\") pod \"maas-api-revoke-sub-a0f9863c-9kglp\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:45.522307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.522285 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0f9863c-9kglp\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:45.523255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.523230 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:13:45.523691 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.523659 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:13:45.558698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.523929 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:13:45.873471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.559906 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:13:45.873722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.831771 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0f9863c-9kglp\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:45.874104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.874069 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnjm\" (UniqueName: \"kubernetes.io/projected/43039834-e511-4d83-b65d-51e6233b1b7a-kube-api-access-6gnjm\") pod \"maas-api-revoke-sub-a0f9863c-9kglp\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:45.874397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.874371 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0f9863c-9kglp\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:45.875794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.875756 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0f9863c-9kglp\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:45.875969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:45.869544 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0f9863c-9kglp\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:46.012140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.012053 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnjm\" (UniqueName: \"kubernetes.io/projected/43039834-e511-4d83-b65d-51e6233b1b7a-kube-api-access-6gnjm\") pod \"maas-api-revoke-sub-a0f9863c-9kglp\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:46.097168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.096820 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:46.101713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.100129 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:13:46.309122 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.309081 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:46.454117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.454078 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:13:46.515386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.456848 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:13:46.515885 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.456889 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:13:46.526245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.465002 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:13:46.526245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.465036 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:13:46.526245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.465054 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:13:46.526245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.465075 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:13:46.526245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.473943 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 02 22:13:46.526245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.339855 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:46.531167 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:46.527754 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405643ba_bf63_46f9_b0c4_d57622186f6a.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:46.531167 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:46.498170 4309 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.331s" Nov 02 22:13:46.531167 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.529139 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:13:46.531167 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.511270 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:13:46.651186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.651149 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:13:46.651802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.651784 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:13:46.651977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.651965 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:13:46.652076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.652066 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp"] Nov 02 22:13:46.652251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.652240 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:13:46.653212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.653191 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:13:46.653602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.653585 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:13:46.660437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.659372 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:13:46.660437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.659445 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:13:46.660437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.659546 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:13:46.660437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.659600 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:13:46.709015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.708956 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc"] Nov 02 22:13:46.726497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.726464 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.732246 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.732221 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc"] Nov 02 22:13:46.759871 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.758948 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq"] Nov 02 22:13:46.767073 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.766649 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.779280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.778051 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq"] Nov 02 22:13:46.779635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.779611 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 22:13:46.779779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.779764 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kzcn8\"" Nov 02 22:13:46.779844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.779818 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 22:13:46.779897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.779856 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 22:13:46.830094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.830011 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.830317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.830298 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.830444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.830426 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvntt\" (UniqueName: \"kubernetes.io/projected/aa204759-fc38-41ee-bc04-f56e56d5de79-kube-api-access-kvntt\") pod \"maas-api-revoke-sub-e0f052da-cqblc\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.830756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.830621 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.830973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.830954 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.831093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.831080 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2512e4-37be-40b8-b5c6-7a6926903034-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.831192 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.831179 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clccs\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-kube-api-access-clccs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.831342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.831326 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.831603 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.831588 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f052da-cqblc\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.831868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.831845 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f052da-cqblc\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.898058 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:46.894230 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43039834_e511_4d83_b65d_51e6233b1b7a.slice/crio-4aa1831e04d5fc1b7b048202b2d7afc398c795ace3a123e82d4127f59e7a3047 WatchSource:0}: Error finding container 4aa1831e04d5fc1b7b048202b2d7afc398c795ace3a123e82d4127f59e7a3047: Status 404 returned error can't find the container with id 4aa1831e04d5fc1b7b048202b2d7afc398c795ace3a123e82d4127f59e7a3047 Nov 02 22:13:46.934959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.934900 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" event={"ID":"43039834-e511-4d83-b65d-51e6233b1b7a","Type":"ContainerStarted","Data":"4aa1831e04d5fc1b7b048202b2d7afc398c795ace3a123e82d4127f59e7a3047"} Nov 02 22:13:46.935126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.934951 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2512e4-37be-40b8-b5c6-7a6926903034-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.935260 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.935237 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clccs\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-kube-api-access-clccs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.935429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.935410 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.935748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.935710 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f052da-cqblc\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.935866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.935849 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f052da-cqblc\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.936112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.936085 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.936211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.936148 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.936211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.936190 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvntt\" (UniqueName: \"kubernetes.io/projected/aa204759-fc38-41ee-bc04-f56e56d5de79-kube-api-access-kvntt\") pod \"maas-api-revoke-sub-e0f052da-cqblc\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.936394 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.936366 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.936505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.936485 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.936655 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.936542 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f052da-cqblc\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.937154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.937129 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.940013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.939985 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.941099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.941075 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2512e4-37be-40b8-b5c6-7a6926903034-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.950765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.943274 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f052da-cqblc\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.953773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.952298 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.953773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.952465 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.953773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.953342 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clccs\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-kube-api-access-clccs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:46.959040 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.958988 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvntt\" (UniqueName: \"kubernetes.io/projected/aa204759-fc38-41ee-bc04-f56e56d5de79-kube-api-access-kvntt\") pod \"maas-api-revoke-sub-e0f052da-cqblc\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:46.965572 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.965531 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp"] Nov 02 22:13:46.965974 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:46.965945 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fm9kq\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:47.063890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:47.063856 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:47.089031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:47.088004 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:47.483056 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:47.483016 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa204759_fc38_41ee_bc04_f56e56d5de79.slice/crio-100b015e6fa2aeada91c9dd7ba279dc47e5f2066c9dd3a1c5ae3a74b7734be3c WatchSource:0}: Error finding container 100b015e6fa2aeada91c9dd7ba279dc47e5f2066c9dd3a1c5ae3a74b7734be3c: Status 404 returned error can't find the container with id 100b015e6fa2aeada91c9dd7ba279dc47e5f2066c9dd3a1c5ae3a74b7734be3c Nov 02 22:13:47.486217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:47.486097 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc"] Nov 02 22:13:47.494690 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:47.494418 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq"] Nov 02 22:13:48.067354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:48.063570 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" event={"ID":"0e2512e4-37be-40b8-b5c6-7a6926903034","Type":"ContainerStarted","Data":"20801108b4ec63cce905546aea96084e32d9f27415336ab436cf1588449836b3"} Nov 02 22:13:48.080630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:48.080594 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" event={"ID":"aa204759-fc38-41ee-bc04-f56e56d5de79","Type":"ContainerStarted","Data":"100b015e6fa2aeada91c9dd7ba279dc47e5f2066c9dd3a1c5ae3a74b7734be3c"} Nov 02 22:13:48.084506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:48.083143 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" event={"ID":"43039834-e511-4d83-b65d-51e6233b1b7a","Type":"ContainerStarted","Data":"cbc1f5b1b9798cf0bfbaacf4891edf1c4461fa4c17279a8a5af48bd72674d734"} Nov 02 22:13:48.112347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:48.112224 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" podStartSLOduration=9.11220387 podStartE2EDuration="9.11220387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:48.102368607 +0000 UTC m=+956.040481841" watchObservedRunningTime="2026-11-02 22:13:48.11220387 +0000 UTC m=+956.050317088" Nov 02 22:13:49.099409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:49.099367 4309 generic.go:356] "Generic (PLEG): container finished" podID="aa204759-fc38-41ee-bc04-f56e56d5de79" containerID="3d372a92d354b0266c47f8370a2aa1a694ff25b08f4ae77c4a7402e7559a403c" exitCode=0 Nov 02 22:13:49.100060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:49.099451 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" event={"ID":"aa204759-fc38-41ee-bc04-f56e56d5de79","Type":"ContainerDied","Data":"3d372a92d354b0266c47f8370a2aa1a694ff25b08f4ae77c4a7402e7559a403c"} Nov 02 22:13:49.101913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:49.101884 4309 generic.go:356] "Generic (PLEG): container finished" podID="43039834-e511-4d83-b65d-51e6233b1b7a" containerID="cbc1f5b1b9798cf0bfbaacf4891edf1c4461fa4c17279a8a5af48bd72674d734" exitCode=0 Nov 02 22:13:49.102074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:49.102052 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" event={"ID":"43039834-e511-4d83-b65d-51e6233b1b7a","Type":"ContainerDied","Data":"cbc1f5b1b9798cf0bfbaacf4891edf1c4461fa4c17279a8a5af48bd72674d734"} Nov 02 22:13:50.369174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.369125 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:50.443012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.442967 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:50.448254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.448106 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-openshift-ca\") pod \"43039834-e511-4d83-b65d-51e6233b1b7a\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " Nov 02 22:13:50.449799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.448512 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-openshift-ca" pod "43039834-e511-4d83-b65d-51e6233b1b7a" (UID: "43039834-e511-4d83-b65d-51e6233b1b7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:50.449799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.448578 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-xks-ca\") pod \"43039834-e511-4d83-b65d-51e6233b1b7a\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " Nov 02 22:13:50.449799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.448748 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43039834-e511-4d83-b65d-51e6233b1b7a-kube-api-access-6gnjm\" (UniqueName: \"kubernetes.io/projected/43039834-e511-4d83-b65d-51e6233b1b7a-kube-api-access-6gnjm\") pod \"43039834-e511-4d83-b65d-51e6233b1b7a\" (UID: \"43039834-e511-4d83-b65d-51e6233b1b7a\") " Nov 02 22:13:50.449799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.449460 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:50.453604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.453576 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-xks-ca" pod "43039834-e511-4d83-b65d-51e6233b1b7a" (UID: "43039834-e511-4d83-b65d-51e6233b1b7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:50.455369 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.455310 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43039834-e511-4d83-b65d-51e6233b1b7a-kube-api-access-6gnjm" pod "43039834-e511-4d83-b65d-51e6233b1b7a" (UID: "43039834-e511-4d83-b65d-51e6233b1b7a"). InnerVolumeSpecName "kube-api-access-6gnjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:50.550891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.550850 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-openshift-ca\") pod \"aa204759-fc38-41ee-bc04-f56e56d5de79\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " Nov 02 22:13:50.551090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.551030 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-xks-ca\") pod \"aa204759-fc38-41ee-bc04-f56e56d5de79\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " Nov 02 22:13:50.551200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.551159 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa204759-fc38-41ee-bc04-f56e56d5de79-kube-api-access-kvntt\" (UniqueName: \"kubernetes.io/projected/aa204759-fc38-41ee-bc04-f56e56d5de79-kube-api-access-kvntt\") pod \"aa204759-fc38-41ee-bc04-f56e56d5de79\" (UID: \"aa204759-fc38-41ee-bc04-f56e56d5de79\") " Nov 02 22:13:50.552264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.552226 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43039834-e511-4d83-b65d-51e6233b1b7a-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:50.552566 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.552270 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gnjm\" (UniqueName: \"kubernetes.io/projected/43039834-e511-4d83-b65d-51e6233b1b7a-kube-api-access-6gnjm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:50.557902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.557868 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-openshift-ca" pod "aa204759-fc38-41ee-bc04-f56e56d5de79" (UID: "aa204759-fc38-41ee-bc04-f56e56d5de79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:50.575012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.574405 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa204759-fc38-41ee-bc04-f56e56d5de79-kube-api-access-kvntt" pod "aa204759-fc38-41ee-bc04-f56e56d5de79" (UID: "aa204759-fc38-41ee-bc04-f56e56d5de79"). InnerVolumeSpecName "kube-api-access-kvntt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:50.575012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.574469 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-xks-ca" pod "aa204759-fc38-41ee-bc04-f56e56d5de79" (UID: "aa204759-fc38-41ee-bc04-f56e56d5de79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:50.651808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.651723 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg"] Nov 02 22:13:50.652707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.652664 4309 cpu_manager.go:414] "containerMap: removing container" podUID="43039834-e511-4d83-b65d-51e6233b1b7a" containerName="revoke-keys" Nov 02 22:13:50.652707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.652709 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43039834-e511-4d83-b65d-51e6233b1b7a" containerName="revoke-keys" Nov 02 22:13:50.652951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.652769 4309 cpu_manager.go:414] "containerMap: removing container" podUID="aa204759-fc38-41ee-bc04-f56e56d5de79" containerName="revoke-keys" Nov 02 22:13:50.652951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.652774 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa204759-fc38-41ee-bc04-f56e56d5de79" containerName="revoke-keys" Nov 02 22:13:50.660727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.657551 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:50.660727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.657584 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa204759-fc38-41ee-bc04-f56e56d5de79-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:50.660727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.657600 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvntt\" (UniqueName: \"kubernetes.io/projected/aa204759-fc38-41ee-bc04-f56e56d5de79-kube-api-access-kvntt\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:50.660727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.658057 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa204759-fc38-41ee-bc04-f56e56d5de79" containerName="revoke-keys" Nov 02 22:13:50.660727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.658085 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="43039834-e511-4d83-b65d-51e6233b1b7a" containerName="revoke-keys" Nov 02 22:13:50.668248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.668219 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.676355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.676321 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg"] Nov 02 22:13:50.759131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.759091 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e219831-eda8-475a-a489-10ed78191791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.759305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.759286 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e219831-eda8-475a-a489-10ed78191791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.759383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.759365 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9599\" (UniqueName: \"kubernetes.io/projected/7e219831-eda8-475a-a489-10ed78191791-kube-api-access-h9599\") pod \"maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.861395 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.861346 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e219831-eda8-475a-a489-10ed78191791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.861557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.861449 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9599\" (UniqueName: \"kubernetes.io/projected/7e219831-eda8-475a-a489-10ed78191791-kube-api-access-h9599\") pod \"maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.861612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.861591 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e219831-eda8-475a-a489-10ed78191791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.862022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.862000 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e219831-eda8-475a-a489-10ed78191791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.864646 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.864614 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e219831-eda8-475a-a489-10ed78191791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.870053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:50.870033 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9599\" (UniqueName: \"kubernetes.io/projected/7e219831-eda8-475a-a489-10ed78191791-kube-api-access-h9599\") pod \"maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:50.921323 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:13:50.921235 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc45e5db9_eea7_43cf_ba46_1595d9511182.slice/crio-b9b296ea7fecb148d0b8b8368c4970c8bcc5f07e3cb46790a80c8f29efc06b9b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice/crio-aec50c45ff46aead4f11a67eaad817f009b8181c5a47f6f06c8b78b4ad2615b7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c986e4_40d6_4841_9e86_cef998f99160.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:13:51.043701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.042996 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:51.164694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.158625 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" event={"ID":"aa204759-fc38-41ee-bc04-f56e56d5de79","Type":"ContainerDied","Data":"100b015e6fa2aeada91c9dd7ba279dc47e5f2066c9dd3a1c5ae3a74b7734be3c"} Nov 02 22:13:51.164694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.158718 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="100b015e6fa2aeada91c9dd7ba279dc47e5f2066c9dd3a1c5ae3a74b7734be3c" Nov 02 22:13:51.164694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.158802 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc" Nov 02 22:13:51.164694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.164280 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" event={"ID":"43039834-e511-4d83-b65d-51e6233b1b7a","Type":"ContainerDied","Data":"4aa1831e04d5fc1b7b048202b2d7afc398c795ace3a123e82d4127f59e7a3047"} Nov 02 22:13:51.164694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.164340 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aa1831e04d5fc1b7b048202b2d7afc398c795ace3a123e82d4127f59e7a3047" Nov 02 22:13:51.164694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.164444 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp" Nov 02 22:13:51.531131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.530601 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg"] Nov 02 22:13:51.538495 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:51.538466 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e219831_eda8_475a_a489_10ed78191791.slice/crio-a1db93c5a0022837845b9e674e4527250a62f474be6df6b78a734fb5ae14a30a WatchSource:0}: Error finding container a1db93c5a0022837845b9e674e4527250a62f474be6df6b78a734fb5ae14a30a: Status 404 returned error can't find the container with id a1db93c5a0022837845b9e674e4527250a62f474be6df6b78a734fb5ae14a30a Nov 02 22:13:51.928704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.928652 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:51.928824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:51.928742 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:52.195924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:52.195851 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:52.195924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:52.195905 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" event={"ID":"0e2512e4-37be-40b8-b5c6-7a6926903034","Type":"ContainerStarted","Data":"90249dafc76a07dc349444f9f28377aab00d2fe0f2841494e28238c97427e361"} Nov 02 22:13:52.196102 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:52.195929 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" event={"ID":"7e219831-eda8-475a-a489-10ed78191791","Type":"ContainerStarted","Data":"ffc7844bdaf765fad3247bc25b0c64a0a7be1d82499054109f01f535e241d2d1"} Nov 02 22:13:52.196102 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:52.195953 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" event={"ID":"7e219831-eda8-475a-a489-10ed78191791","Type":"ContainerStarted","Data":"a1db93c5a0022837845b9e674e4527250a62f474be6df6b78a734fb5ae14a30a"} Nov 02 22:13:52.206607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:52.206547 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" podStartSLOduration=2.206529925 podStartE2EDuration="2.206529925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:52.20376881 +0000 UTC m=+960.141882041" watchObservedRunningTime="2026-11-02 22:13:52.206529925 +0000 UTC m=+960.144643144" Nov 02 22:13:52.221843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:52.221790 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" podStartSLOduration=2.8086447420000002 podStartE2EDuration="7.221772743s" totalImagesPullingTime="4.413128001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:52.220354198 +0000 UTC m=+960.158467417" watchObservedRunningTime="2026-11-02 22:13:52.221772743 +0000 UTC m=+960.159885964" Nov 02 22:13:53.217902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:53.217807 4309 generic.go:356] "Generic (PLEG): container finished" podID="7e219831-eda8-475a-a489-10ed78191791" containerID="ffc7844bdaf765fad3247bc25b0c64a0a7be1d82499054109f01f535e241d2d1" exitCode=0 Nov 02 22:13:53.225447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:53.225230 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" event={"ID":"7e219831-eda8-475a-a489-10ed78191791","Type":"ContainerDied","Data":"ffc7844bdaf765fad3247bc25b0c64a0a7be1d82499054109f01f535e241d2d1"} Nov 02 22:13:53.234628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:53.231015 4309 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fm9kq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:13:53.234628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:53.231067 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" podUID="0e2512e4-37be-40b8-b5c6-7a6926903034" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:13:54.495607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.495543 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:54.605161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.605134 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e219831-eda8-475a-a489-10ed78191791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e219831-eda8-475a-a489-10ed78191791-maas-api-xks-ca\") pod \"7e219831-eda8-475a-a489-10ed78191791\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " Nov 02 22:13:54.605457 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.605440 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e219831-eda8-475a-a489-10ed78191791-kube-api-access-h9599\" (UniqueName: \"kubernetes.io/projected/7e219831-eda8-475a-a489-10ed78191791-kube-api-access-h9599\") pod \"7e219831-eda8-475a-a489-10ed78191791\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " Nov 02 22:13:54.605643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.605627 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e219831-eda8-475a-a489-10ed78191791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e219831-eda8-475a-a489-10ed78191791-maas-api-openshift-ca\") pod \"7e219831-eda8-475a-a489-10ed78191791\" (UID: \"7e219831-eda8-475a-a489-10ed78191791\") " Nov 02 22:13:54.606024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.605994 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e219831-eda8-475a-a489-10ed78191791-maas-api-openshift-ca" pod "7e219831-eda8-475a-a489-10ed78191791" (UID: "7e219831-eda8-475a-a489-10ed78191791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:13:54.606783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.606760 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e219831-eda8-475a-a489-10ed78191791-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:54.607595 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.607569 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e219831-eda8-475a-a489-10ed78191791-maas-api-xks-ca" pod "7e219831-eda8-475a-a489-10ed78191791" (UID: "7e219831-eda8-475a-a489-10ed78191791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:13:54.607716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.607659 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e219831-eda8-475a-a489-10ed78191791-kube-api-access-h9599" pod "7e219831-eda8-475a-a489-10ed78191791" (UID: "7e219831-eda8-475a-a489-10ed78191791"). InnerVolumeSpecName "kube-api-access-h9599". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:13:54.708405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.708367 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9599\" (UniqueName: \"kubernetes.io/projected/7e219831-eda8-475a-a489-10ed78191791-kube-api-access-h9599\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:54.708574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:54.708410 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e219831-eda8-475a-a489-10ed78191791-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:13:55.243451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:55.243424 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" Nov 02 22:13:55.243622 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:55.243460 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg" event={"ID":"7e219831-eda8-475a-a489-10ed78191791","Type":"ContainerDied","Data":"a1db93c5a0022837845b9e674e4527250a62f474be6df6b78a734fb5ae14a30a"} Nov 02 22:13:55.243622 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:55.243507 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1db93c5a0022837845b9e674e4527250a62f474be6df6b78a734fb5ae14a30a" Nov 02 22:13:56.222408 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.222375 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:13:56.665757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.665665 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl"] Nov 02 22:13:56.667325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.667291 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7e219831-eda8-475a-a489-10ed78191791" containerName="revoke-keys" Nov 02 22:13:56.667325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.667323 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e219831-eda8-475a-a489-10ed78191791" containerName="revoke-keys" Nov 02 22:13:56.667975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.667544 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e219831-eda8-475a-a489-10ed78191791" containerName="revoke-keys" Nov 02 22:13:56.678127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.675505 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.701212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.685800 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-10852f9c-openshift-default-dockercfg-2znr9\"" Nov 02 22:13:56.707350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.707225 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl"] Nov 02 22:13:56.730650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.724287 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-podinfo\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.730650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.724356 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-socket\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.730650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.724413 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvn2p\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-kube-api-access-jvn2p\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.730650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.724489 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-envoy\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.730650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.724556 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-data\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.730650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.724602 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-certs\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.730650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.724642 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-token\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.730650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.724767 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2972de31-06c1-4022-a3ab-3796c5ecd136-istiod-ca-cert\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.730650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.724800 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-credential-socket\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.831706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.826856 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2972de31-06c1-4022-a3ab-3796c5ecd136-istiod-ca-cert\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.831706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.827864 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2972de31-06c1-4022-a3ab-3796c5ecd136-istiod-ca-cert\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.831706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.827940 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-credential-socket\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.831706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.828140 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-podinfo\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.831706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.828194 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-socket\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.831706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.828264 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvn2p\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-kube-api-access-jvn2p\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.831706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.829348 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-socket\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.831706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.830478 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-credential-socket\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.844491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.836854 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-envoy\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.844491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.836944 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-data\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.844491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.837064 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-certs\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.844491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.837097 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-token\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.844491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.841558 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-data\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.844491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.841874 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-certs\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.870158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.859962 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-envoy\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.870158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.862342 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-podinfo\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.896940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.896905 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvn2p\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-kube-api-access-jvn2p\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:56.905697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:56.900844 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-token\") pod \"e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:57.041645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.040347 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:57.376387 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:57.376358 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2972de31_06c1_4022_a3ab_3796c5ecd136.slice/crio-1bfaf9281091b5ea3b5a57db0b09c53ab9e763b708adbabda91d5966bc61d8a6 WatchSource:0}: Error finding container 1bfaf9281091b5ea3b5a57db0b09c53ab9e763b708adbabda91d5966bc61d8a6: Status 404 returned error can't find the container with id 1bfaf9281091b5ea3b5a57db0b09c53ab9e763b708adbabda91d5966bc61d8a6 Nov 02 22:13:57.376821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.376791 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl"] Nov 02 22:13:57.380002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.379967 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:57.380113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.380036 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:57.380113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.380071 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:13:57.665317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.665230 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf"] Nov 02 22:13:57.676708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.673839 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:57.688699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.685721 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf"] Nov 02 22:13:57.705696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.698259 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:13:57.705696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.698311 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:13:57.733585 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.733511 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:13:57.733585 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.733532 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:13:57.759930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.759897 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqds\" (UniqueName: \"kubernetes.io/projected/1dc68026-e119-4d93-addc-9bb44e9dbe9f-kube-api-access-rdqds\") pod \"maas-api-revoke-sub-56138600-8gwbf\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:57.763140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.760807 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56138600-8gwbf\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:57.763140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.760863 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56138600-8gwbf\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:57.862845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.862811 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56138600-8gwbf\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:57.863028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.862867 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56138600-8gwbf\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:57.863028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.863008 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqds\" (UniqueName: \"kubernetes.io/projected/1dc68026-e119-4d93-addc-9bb44e9dbe9f-kube-api-access-rdqds\") pod \"maas-api-revoke-sub-56138600-8gwbf\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:57.863591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.863567 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56138600-8gwbf\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:57.865159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.865138 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56138600-8gwbf\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:57.870525 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:57.870504 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqds\" (UniqueName: \"kubernetes.io/projected/1dc68026-e119-4d93-addc-9bb44e9dbe9f-kube-api-access-rdqds\") pod \"maas-api-revoke-sub-56138600-8gwbf\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:58.012456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:58.012379 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:13:58.293243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:58.293144 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" event={"ID":"2972de31-06c1-4022-a3ab-3796c5ecd136","Type":"ContainerStarted","Data":"51189d2b30d4cd50e2c713d1a0f4a1749e3201feed477b5ebeca4e3690e69142"} Nov 02 22:13:58.293243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:58.293213 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" event={"ID":"2972de31-06c1-4022-a3ab-3796c5ecd136","Type":"ContainerStarted","Data":"1bfaf9281091b5ea3b5a57db0b09c53ab9e763b708adbabda91d5966bc61d8a6"} Nov 02 22:13:58.312833 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:13:58.312799 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc68026_e119_4d93_addc_9bb44e9dbe9f.slice/crio-087b13d4e94cec26f919b8c327297c5661b7e8251841c13063f2cdbe824f8b42 WatchSource:0}: Error finding container 087b13d4e94cec26f919b8c327297c5661b7e8251841c13063f2cdbe824f8b42: Status 404 returned error can't find the container with id 087b13d4e94cec26f919b8c327297c5661b7e8251841c13063f2cdbe824f8b42 Nov 02 22:13:58.313569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:58.313536 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf"] Nov 02 22:13:58.316968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:58.316929 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" podStartSLOduration=2.31691629 podStartE2EDuration="2.31691629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:13:58.311812188 +0000 UTC m=+966.249925408" watchObservedRunningTime="2026-11-02 22:13:58.31691629 +0000 UTC m=+966.255029507" Nov 02 22:13:59.043069 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:59.042983 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:59.048650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:59.048626 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:59.316411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:59.316378 4309 generic.go:356] "Generic (PLEG): container finished" podID="1dc68026-e119-4d93-addc-9bb44e9dbe9f" containerID="aeed214d63fa011f853df5a1e337906a41ac2566ab10e2fc40b7b6e831f72c0f" exitCode=0 Nov 02 22:13:59.316852 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:59.316821 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" event={"ID":"1dc68026-e119-4d93-addc-9bb44e9dbe9f","Type":"ContainerDied","Data":"aeed214d63fa011f853df5a1e337906a41ac2566ab10e2fc40b7b6e831f72c0f"} Nov 02 22:13:59.318608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:59.318580 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:13:59.318787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:59.318757 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" event={"ID":"1dc68026-e119-4d93-addc-9bb44e9dbe9f","Type":"ContainerStarted","Data":"087b13d4e94cec26f919b8c327297c5661b7e8251841c13063f2cdbe824f8b42"} Nov 02 22:13:59.318889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:13:59.318832 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:14:00.552551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.552524 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:14:00.647529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.647444 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-openshift-ca\") pod \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " Nov 02 22:14:00.647529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.647504 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-xks-ca\") pod \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " Nov 02 22:14:00.647878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.647804 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc68026-e119-4d93-addc-9bb44e9dbe9f-kube-api-access-rdqds\" (UniqueName: \"kubernetes.io/projected/1dc68026-e119-4d93-addc-9bb44e9dbe9f-kube-api-access-rdqds\") pod \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\" (UID: \"1dc68026-e119-4d93-addc-9bb44e9dbe9f\") " Nov 02 22:14:00.647878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.647848 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-openshift-ca" pod "1dc68026-e119-4d93-addc-9bb44e9dbe9f" (UID: "1dc68026-e119-4d93-addc-9bb44e9dbe9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:00.648719 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.648690 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:00.650594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.650572 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc68026-e119-4d93-addc-9bb44e9dbe9f-kube-api-access-rdqds" pod "1dc68026-e119-4d93-addc-9bb44e9dbe9f" (UID: "1dc68026-e119-4d93-addc-9bb44e9dbe9f"). InnerVolumeSpecName "kube-api-access-rdqds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:00.657035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.656981 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-xks-ca" pod "1dc68026-e119-4d93-addc-9bb44e9dbe9f" (UID: "1dc68026-e119-4d93-addc-9bb44e9dbe9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:00.791778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.791696 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc68026-e119-4d93-addc-9bb44e9dbe9f-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:00.791778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.791735 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdqds\" (UniqueName: \"kubernetes.io/projected/1dc68026-e119-4d93-addc-9bb44e9dbe9f-kube-api-access-rdqds\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:00.884511 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:00.884463 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn"] Nov 02 22:14:01.414560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.414530 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" Nov 02 22:14:01.414981 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.414822 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf" event={"ID":"1dc68026-e119-4d93-addc-9bb44e9dbe9f","Type":"ContainerDied","Data":"087b13d4e94cec26f919b8c327297c5661b7e8251841c13063f2cdbe824f8b42"} Nov 02 22:14:01.415075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.415008 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="087b13d4e94cec26f919b8c327297c5661b7e8251841c13063f2cdbe824f8b42" Nov 02 22:14:01.638721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.635299 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n"] Nov 02 22:14:01.638721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.636208 4309 cpu_manager.go:414] "containerMap: removing container" podUID="1dc68026-e119-4d93-addc-9bb44e9dbe9f" containerName="revoke-keys" Nov 02 22:14:01.638721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.636225 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc68026-e119-4d93-addc-9bb44e9dbe9f" containerName="revoke-keys" Nov 02 22:14:01.638721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.636424 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc68026-e119-4d93-addc-9bb44e9dbe9f" containerName="revoke-keys" Nov 02 22:14:01.653103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.641439 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:01.653103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.649209 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n"] Nov 02 22:14:01.653103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.650985 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6a72dd63-dockercfg-hwxxs\"" Nov 02 22:14:01.653103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.651104 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6a72dd63\"" Nov 02 22:14:01.756440 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.756346 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f299ed-7957-4ade-9951-36e5f02ecc0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n\" (UID: \"27f299ed-7957-4ade-9951-36e5f02ecc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:01.759141 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.759114 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzpn\" (UniqueName: \"kubernetes.io/projected/27f299ed-7957-4ade-9951-36e5f02ecc0c-kube-api-access-ltzpn\") pod \"payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n\" (UID: \"27f299ed-7957-4ade-9951-36e5f02ecc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:01.881714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.880457 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f299ed-7957-4ade-9951-36e5f02ecc0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n\" (UID: \"27f299ed-7957-4ade-9951-36e5f02ecc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:01.881714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.880515 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzpn\" (UniqueName: \"kubernetes.io/projected/27f299ed-7957-4ade-9951-36e5f02ecc0c-kube-api-access-ltzpn\") pod \"payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n\" (UID: \"27f299ed-7957-4ade-9951-36e5f02ecc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:01.882649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.882600 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f299ed-7957-4ade-9951-36e5f02ecc0c-plugins-config-volume\") pod \"payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n\" (UID: \"27f299ed-7957-4ade-9951-36e5f02ecc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:01.907418 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.907382 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzpn\" (UniqueName: \"kubernetes.io/projected/27f299ed-7957-4ade-9951-36e5f02ecc0c-kube-api-access-ltzpn\") pod \"payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n\" (UID: \"27f299ed-7957-4ade-9951-36e5f02ecc0c\") " pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:01.979335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:01.979291 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:02.544025 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:02.543993 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n"] Nov 02 22:14:03.450857 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:03.450820 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" event={"ID":"27f299ed-7957-4ade-9951-36e5f02ecc0c","Type":"ContainerStarted","Data":"44cdc84e59b9e2d37ec1e26ef2fd8d8215866f6ff4ef709058bd16401654dd13"} Nov 02 22:14:03.453594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:03.450877 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" event={"ID":"27f299ed-7957-4ade-9951-36e5f02ecc0c","Type":"ContainerStarted","Data":"8d09600f5aec392fb8606072b8bd708c1b17047b17001b843f21f11993b02ac5"} Nov 02 22:14:03.453594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:03.451905 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:03.477367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:03.477313 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" podStartSLOduration=2.477295996 podStartE2EDuration="2.477295996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:03.46972411 +0000 UTC m=+971.407837332" watchObservedRunningTime="2026-11-02 22:14:03.477295996 +0000 UTC m=+971.415409246" Nov 02 22:14:03.575567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:03.575536 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg"] Nov 02 22:14:03.583650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:03.583609 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec62fcf4-jt7vg"] Nov 02 22:14:04.194042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:04.194005 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e219831-eda8-475a-a489-10ed78191791" path="/var/lib/kubelet/pods/7e219831-eda8-475a-a489-10ed78191791/volumes" Nov 02 22:14:05.040927 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.040892 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp"] Nov 02 22:14:05.046561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.046429 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.059062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.054941 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bb16f94c-gw-openshift-default-dockercfg-2qhp8\"" Nov 02 22:14:05.066267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.066242 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp"] Nov 02 22:14:05.193897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.193854 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-certs\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.194090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.193977 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istiod-ca-cert\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.194090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.194034 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-credential-socket\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.194090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.194073 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqmc\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-kube-api-access-hjqmc\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.194250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.194221 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-data\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.194301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.194286 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-envoy\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.194371 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.194355 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-socket\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.194447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.194428 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-token\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.194537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.194518 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-podinfo\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.326551 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-envoy\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.326691 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-socket\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.326796 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-token\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.326942 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-podinfo\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.327170 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-certs\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.327317 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istiod-ca-cert\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.327412 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-credential-socket\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.327504 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqmc\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-kube-api-access-hjqmc\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.327617 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-data\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.357696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.348213 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-socket\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.357696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.350881 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istiod-ca-cert\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.357696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.351470 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-credential-socket\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.357696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.353046 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-certs\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.357696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.353285 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-data\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.386612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.373160 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-token\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.396696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.390335 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqmc\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-kube-api-access-hjqmc\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.396869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.396719 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-podinfo\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.433697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.433152 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-envoy\") pod \"e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.677794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.676952 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:05.892401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:05.885360 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" podUID="7bd197b7-cb3d-4c50-95e1-f758cd47af00" containerName="payload-processing" containerID="cri-o://f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6" gracePeriod=25 Nov 02 22:14:06.045294 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:14:06.045264 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00ef6bf_24d1_4cfb_bc8b_9947b36a7a19.slice/crio-fd082032f8b8c9da069caf9362eb91a25d1e38a076b81e376a9e061390ee1a81 WatchSource:0}: Error finding container fd082032f8b8c9da069caf9362eb91a25d1e38a076b81e376a9e061390ee1a81: Status 404 returned error can't find the container with id fd082032f8b8c9da069caf9362eb91a25d1e38a076b81e376a9e061390ee1a81 Nov 02 22:14:06.048446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.048421 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp"] Nov 02 22:14:06.048905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.048873 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:14:06.048963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.048941 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:14:06.049003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.048969 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:14:06.312462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.309253 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:14:06.441059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.440987 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd197b7-cb3d-4c50-95e1-f758cd47af00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd197b7-cb3d-4c50-95e1-f758cd47af00-plugins-config-volume\") pod \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\" (UID: \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\") " Nov 02 22:14:06.441198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.441079 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd197b7-cb3d-4c50-95e1-f758cd47af00-kube-api-access-kz64s\" (UniqueName: \"kubernetes.io/projected/7bd197b7-cb3d-4c50-95e1-f758cd47af00-kube-api-access-kz64s\") pod \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\" (UID: \"7bd197b7-cb3d-4c50-95e1-f758cd47af00\") " Nov 02 22:14:06.441423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.441394 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd197b7-cb3d-4c50-95e1-f758cd47af00-plugins-config-volume" pod "7bd197b7-cb3d-4c50-95e1-f758cd47af00" (UID: "7bd197b7-cb3d-4c50-95e1-f758cd47af00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:06.441748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.441731 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd197b7-cb3d-4c50-95e1-f758cd47af00-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:06.443322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.443299 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd197b7-cb3d-4c50-95e1-f758cd47af00-kube-api-access-kz64s" pod "7bd197b7-cb3d-4c50-95e1-f758cd47af00" (UID: "7bd197b7-cb3d-4c50-95e1-f758cd47af00"). InnerVolumeSpecName "kube-api-access-kz64s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:06.542810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.542756 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz64s\" (UniqueName: \"kubernetes.io/projected/7bd197b7-cb3d-4c50-95e1-f758cd47af00-kube-api-access-kz64s\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:06.543444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.543417 4309 generic.go:356] "Generic (PLEG): container finished" podID="7bd197b7-cb3d-4c50-95e1-f758cd47af00" containerID="f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6" exitCode=0 Nov 02 22:14:06.543543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.543459 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" Nov 02 22:14:06.543543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.543501 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" event={"ID":"7bd197b7-cb3d-4c50-95e1-f758cd47af00","Type":"ContainerDied","Data":"f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6"} Nov 02 22:14:06.543642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.543559 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn" event={"ID":"7bd197b7-cb3d-4c50-95e1-f758cd47af00","Type":"ContainerDied","Data":"6f3661526223ce484a03210c3024eb1975e7ccdfe40f365b720acf3bdbe65cbe"} Nov 02 22:14:06.543642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.543591 4309 scope.go:122] "RemoveContainer" containerID="f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6" Nov 02 22:14:06.546024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.545997 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" event={"ID":"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19","Type":"ContainerStarted","Data":"c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444"} Nov 02 22:14:06.546133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.546053 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" event={"ID":"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19","Type":"ContainerStarted","Data":"fd082032f8b8c9da069caf9362eb91a25d1e38a076b81e376a9e061390ee1a81"} Nov 02 22:14:06.557998 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.557983 4309 scope.go:122] "RemoveContainer" containerID="f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6" Nov 02 22:14:06.558285 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:14:06.558266 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6\": container with ID starting with f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6 not found: ID does not exist" containerID="f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6" Nov 02 22:14:06.558350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.558296 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6"} err="failed to get container status \"f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6\": rpc error: code = NotFound desc = could not find container \"f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6\": container with ID starting with f6464c7ce6f6ff36531c7779ed286c5aaca93d67027a29f431f0f79c17296fe6 not found: ID does not exist" Nov 02 22:14:06.570249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.570202 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" podStartSLOduration=1.570186847 podStartE2EDuration="1.570186847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:06.564467311 +0000 UTC m=+974.502580530" watchObservedRunningTime="2026-11-02 22:14:06.570186847 +0000 UTC m=+974.508300066" Nov 02 22:14:06.597834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.597806 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn"] Nov 02 22:14:06.606715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.606661 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec62fcf4-7bb59b8497-ql9qn"] Nov 02 22:14:06.679406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:06.679328 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:07.684995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.684960 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:07.876060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.876027 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk"] Nov 02 22:14:07.876862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.876844 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7bd197b7-cb3d-4c50-95e1-f758cd47af00" containerName="payload-processing" Nov 02 22:14:07.876862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.876861 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd197b7-cb3d-4c50-95e1-f758cd47af00" containerName="payload-processing" Nov 02 22:14:07.877037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.877028 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd197b7-cb3d-4c50-95e1-f758cd47af00" containerName="payload-processing" Nov 02 22:14:07.900520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.900486 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:07.904475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.904177 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:07.904475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.904253 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:07.904475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.904412 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:14:07.904475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.904458 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:07.906408 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.906377 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk"] Nov 02 22:14:07.968195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.968120 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-965669d9-zxqrk\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:07.968195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.968175 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-965669d9-zxqrk\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:07.968389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:07.968348 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhhf\" (UniqueName: \"kubernetes.io/projected/221fb35c-8484-4a36-93a1-4ad9904ceee8-kube-api-access-6dhhf\") pod \"maas-api-revoke-sub-965669d9-zxqrk\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:08.069963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.069929 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-965669d9-zxqrk\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:08.070146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.070038 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhhf\" (UniqueName: \"kubernetes.io/projected/221fb35c-8484-4a36-93a1-4ad9904ceee8-kube-api-access-6dhhf\") pod \"maas-api-revoke-sub-965669d9-zxqrk\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:08.070281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.070263 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-965669d9-zxqrk\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:08.070764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.070738 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-965669d9-zxqrk\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:08.072649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.072620 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-965669d9-zxqrk\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:08.077841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.077819 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhhf\" (UniqueName: \"kubernetes.io/projected/221fb35c-8484-4a36-93a1-4ad9904ceee8-kube-api-access-6dhhf\") pod \"maas-api-revoke-sub-965669d9-zxqrk\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:08.190590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.190556 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd197b7-cb3d-4c50-95e1-f758cd47af00" path="/var/lib/kubelet/pods/7bd197b7-cb3d-4c50-95e1-f758cd47af00/volumes" Nov 02 22:14:08.236721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.236632 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:08.538369 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:14:08.538183 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221fb35c_8484_4a36_93a1_4ad9904ceee8.slice/crio-49d4c500d4fb5a0c64ed8c6a287aad1530d7ec31cada5586d163e34aee421bad WatchSource:0}: Error finding container 49d4c500d4fb5a0c64ed8c6a287aad1530d7ec31cada5586d163e34aee421bad: Status 404 returned error can't find the container with id 49d4c500d4fb5a0c64ed8c6a287aad1530d7ec31cada5586d163e34aee421bad Nov 02 22:14:08.541867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.541826 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk"] Nov 02 22:14:08.573867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.573835 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" event={"ID":"221fb35c-8484-4a36-93a1-4ad9904ceee8","Type":"ContainerStarted","Data":"49d4c500d4fb5a0c64ed8c6a287aad1530d7ec31cada5586d163e34aee421bad"} Nov 02 22:14:08.574496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.574471 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:08.575441 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:08.575420 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:14:09.590530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:09.590491 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" event={"ID":"221fb35c-8484-4a36-93a1-4ad9904ceee8","Type":"ContainerStarted","Data":"3721229251e4619ac669e4197d4a81324c0dfaeddce64663439155329fb18584"} Nov 02 22:14:09.609586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:09.609515 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" podStartSLOduration=2.609497573 podStartE2EDuration="2.609497573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:09.607193951 +0000 UTC m=+977.545307171" watchObservedRunningTime="2026-11-02 22:14:09.609497573 +0000 UTC m=+977.547610792" Nov 02 22:14:10.602389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:10.602352 4309 generic.go:356] "Generic (PLEG): container finished" podID="221fb35c-8484-4a36-93a1-4ad9904ceee8" containerID="3721229251e4619ac669e4197d4a81324c0dfaeddce64663439155329fb18584" exitCode=0 Nov 02 22:14:10.602872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:10.602469 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" event={"ID":"221fb35c-8484-4a36-93a1-4ad9904ceee8","Type":"ContainerDied","Data":"3721229251e4619ac669e4197d4a81324c0dfaeddce64663439155329fb18584"} Nov 02 22:14:11.605533 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:14:11.605494 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:14:11.825579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:11.825541 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:11.921756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:11.909471 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-openshift-ca\") pod \"221fb35c-8484-4a36-93a1-4ad9904ceee8\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " Nov 02 22:14:11.922886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:11.922726 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-openshift-ca" pod "221fb35c-8484-4a36-93a1-4ad9904ceee8" (UID: "221fb35c-8484-4a36-93a1-4ad9904ceee8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:11.932663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:11.932079 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221fb35c-8484-4a36-93a1-4ad9904ceee8-kube-api-access-6dhhf\" (UniqueName: \"kubernetes.io/projected/221fb35c-8484-4a36-93a1-4ad9904ceee8-kube-api-access-6dhhf\") pod \"221fb35c-8484-4a36-93a1-4ad9904ceee8\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " Nov 02 22:14:11.933006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:11.932986 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-xks-ca\") pod \"221fb35c-8484-4a36-93a1-4ad9904ceee8\" (UID: \"221fb35c-8484-4a36-93a1-4ad9904ceee8\") " Nov 02 22:14:11.940834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:11.940803 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:11.964733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:11.963513 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-xks-ca" pod "221fb35c-8484-4a36-93a1-4ad9904ceee8" (UID: "221fb35c-8484-4a36-93a1-4ad9904ceee8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:11.966949 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:11.966910 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221fb35c-8484-4a36-93a1-4ad9904ceee8-kube-api-access-6dhhf" pod "221fb35c-8484-4a36-93a1-4ad9904ceee8" (UID: "221fb35c-8484-4a36-93a1-4ad9904ceee8"). InnerVolumeSpecName "kube-api-access-6dhhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:12.045611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:12.045563 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dhhf\" (UniqueName: \"kubernetes.io/projected/221fb35c-8484-4a36-93a1-4ad9904ceee8-kube-api-access-6dhhf\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:12.045611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:12.045599 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221fb35c-8484-4a36-93a1-4ad9904ceee8-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:12.641739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:12.641642 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" Nov 02 22:14:12.641739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:12.641655 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk" event={"ID":"221fb35c-8484-4a36-93a1-4ad9904ceee8","Type":"ContainerDied","Data":"49d4c500d4fb5a0c64ed8c6a287aad1530d7ec31cada5586d163e34aee421bad"} Nov 02 22:14:12.642314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:12.641752 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d4c500d4fb5a0c64ed8c6a287aad1530d7ec31cada5586d163e34aee421bad" Nov 02 22:14:15.542745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:15.542279 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:16.917699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:16.915272 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq"] Nov 02 22:14:16.934713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:16.934659 4309 cpu_manager.go:414] "containerMap: removing container" podUID="221fb35c-8484-4a36-93a1-4ad9904ceee8" containerName="revoke-keys" Nov 02 22:14:16.934713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:16.934711 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221fb35c-8484-4a36-93a1-4ad9904ceee8" containerName="revoke-keys" Nov 02 22:14:16.935022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:16.934995 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="221fb35c-8484-4a36-93a1-4ad9904ceee8" containerName="revoke-keys" Nov 02 22:14:17.066563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:16.958788 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.066563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.005902 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:14:17.066563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.007217 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Nov 02 22:14:17.066563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.040720 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.066563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.040841 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.066563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.041022 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6zt\" (UniqueName: \"kubernetes.io/projected/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-kube-api-access-zb6zt\") pod \"maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.131480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.109460 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:17.131480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.109566 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:17.131480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.109765 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:14:17.131480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.109843 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:17.154447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.143700 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6zt\" (UniqueName: \"kubernetes.io/projected/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-kube-api-access-zb6zt\") pod \"maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.154447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.143868 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.154447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.144010 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.166699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.157191 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.290758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.252663 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq"] Nov 02 22:14:17.316701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.299822 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:14:17.316701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.304919 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:14:17.316701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.313921 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:17.344931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.344201 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:17.369862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.355233 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.408877 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.408838 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:14:17.496700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.477338 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6zt\" (UniqueName: \"kubernetes.io/projected/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-kube-api-access-zb6zt\") pod \"maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.497090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.497045 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:17.536607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.536553 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:17.563020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.562983 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:17.785334 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.696822 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:14:17.807701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.786274 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:17.823897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.823798 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:17.824192 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.824157 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:17.912702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:17.907774 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:18.021698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.007540 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:18.021698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.008055 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:14:18.342814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.272834 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:18.342814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.272904 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:18.342814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.314308 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:14:18.342814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.314335 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:14:18.342814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.314304 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:18.342814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.341792 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:18.342814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.341860 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:18.501707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.491171 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:18.597887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.597776 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:18.731702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.730143 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:14:18.754479 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.754435 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:14:18.777236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.777180 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:18.824744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.815046 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:14:18.872181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.850926 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:14:18.883886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.883851 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:14:18.887840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.887812 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:14:18.896847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.896817 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:14:18.897217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.897198 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:14:18.897404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.897390 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:14:18.901722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.897745 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:14:19.365966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.365939 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:14:19.366528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.366507 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:14:19.366744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.366725 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:14:19.399239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:18.913569 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:19.401796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.365808 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:19.469297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.469264 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:14:19.537709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.529844 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:19.742848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.742809 4309 patch_prober.go:29] interesting pod/thanos-querier-75fcc597f8-zw9z8 container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:19.826966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.743629 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xgv2l" podUID="18e45205-8cd3-40c2-b688-6d31bc145c0a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded" Nov 02 22:14:19.876699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.870257 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 02 22:14:19.876699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:19.873380 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:14:20.065801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.021399 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:20.168011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.167967 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:20.169609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.169573 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:14:20.169856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.169831 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" podUID="774397e4-c8ee-4734-8f5a-7befd9b4d9dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded" Nov 02 22:14:20.235732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.221204 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-75fcc597f8-zw9z8" podUID="c9ef1d8c-8213-455c-a988-925d8cdaa2b5" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:20.285294 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.285250 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:14:20.738348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.321739 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" podUID="5b0b2263-76f6-4f5d-b402-be9d5ec3b645" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.47:50052\" within 1s: context deadline exceeded" Nov 02 22:14:20.738348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.347340 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:14:20.738348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.547962 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:14:20.738348 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:14:20.548179 4309 logging.go:55] [core] [Channel #956 SubChannel #958]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.47:50052", ServerName: "10.134.0.47:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.134.0.2:55766->10.134.0.47:50052: use of closed network connection" Nov 02 22:14:20.738348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.705183 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-df7vx" podUID="d2b2a428-093b-4d9c-8b88-ad2b24b447c8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": context deadline exceeded" Nov 02 22:14:20.738348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.705937 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:20.738348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.731782 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:20.752744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.748899 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-hzbck" podUID="774397e4-c8ee-4734-8f5a-7befd9b4d9dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:20.833828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.822495 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:14:20.843702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:20.839723 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:14:21.006889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.006518 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" podUID="2304b2a9-889a-4265-9ff3-25ac877f2f0d" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Nov 02 22:14:21.082739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.061759 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Nov 02 22:14:21.082739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.061949 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Nov 02 22:14:21.144384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.144290 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl" podUID="96e4208b-1bbe-4a26-a5c0-77783d8239b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:21.332733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.330971 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:14:21.337130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.337083 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:14:21.342078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.342033 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:14:21.344741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.342266 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:14:21.561521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.561463 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:21.607261 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:14:21.566792 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2276b6bc_c7f2_43d1_ae6b_87f04c73c197.slice/crio-0998da4ebd6df1d268ae1f73705a7d6d29141005607175fa9185834bdcff0ef5\": RecentStats: unable to find data in memory cache]" Nov 02 22:14:21.661284 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.661239 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:21.662276 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.662227 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:21.670502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.670438 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:21.775469 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.754592 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:21.830390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.789601 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:21.832142 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:14:21.801434 4309 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.519s" Nov 02 22:14:21.835786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.835736 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq"] Nov 02 22:14:21.877794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.862194 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:21.998899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:21.998857 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:14:22.099211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.099105 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" event={"ID":"2276b6bc-c7f2-43d1-ae6b-87f04c73c197","Type":"ContainerStarted","Data":"0998da4ebd6df1d268ae1f73705a7d6d29141005607175fa9185834bdcff0ef5"} Nov 02 22:14:22.365786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.340857 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:14:22.385358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.341812 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:22.386926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.366997 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:14:22.397139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.367601 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:22.625287 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.625198 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:22.625569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.625535 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:22.669942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.669898 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:14:22.715702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.688138 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:14:22.715702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.689282 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:22.715702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.691764 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:22.715702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.691811 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:22.766736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.765776 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:22.884750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.841454 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:23.139767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.864618 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:14:23.139767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:22.958128 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:14:23.139767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.068114 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:23.139767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.100457 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:23.139767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.075881 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:14:23.376138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.370774 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:14:23.421814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.417932 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:14:23.615702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.475540 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:23.741030 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.475729 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:14:23.742112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.690958 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:23.745327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.690997 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:14:23.763652 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.691278 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:23.764163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.739917 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:23.769797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.769763 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:23.773693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.739980 4309 patch_prober.go:29] interesting pod/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:23.773885 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.773859 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:23.838698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.813815 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:23.838994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:23.838953 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:24.144307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.127218 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:24.144307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.129257 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:14:24.144307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.129563 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:14:24.144307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.129776 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:14:24.144307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.129818 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:24.144307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.129836 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:14:24.367871 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.358561 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:24.469850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.391445 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:24.492058 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.391486 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Nov 02 22:14:24.492444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.391801 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:24.704912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.697346 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:24.803735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.784565 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:14:24.803735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.784851 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:14:24.803735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.785334 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:24.803735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.790880 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:24.803735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:24.791834 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:14:25.013445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.013357 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm"] Nov 02 22:14:25.204821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.200391 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:14:25.329817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.206977 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:14:25.329817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.226920 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:14:25.329817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.233639 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:14:25.600764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.573766 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:25.765202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.620010 4309 generic.go:356] "Generic (PLEG): container finished" podID="2276b6bc-c7f2-43d1-ae6b-87f04c73c197" containerID="d9f4717b45e9bdecc417f8fd8b5ad7425621ac342483367d52d67c62922ffd7c" exitCode=0 Nov 02 22:14:25.765509 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.739800 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:25.793412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.793371 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:25.805739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.805697 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:14:25.814087 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.808085 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:14:25.814608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.814577 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:14:25.819775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:25.819741 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:14:26.285700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.284425 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:26.285700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.284716 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:14:26.285700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.284939 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:14:26.285700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.285105 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:14:26.382439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.378202 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm"] Nov 02 22:14:26.382439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.378254 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" event={"ID":"2276b6bc-c7f2-43d1-ae6b-87f04c73c197","Type":"ContainerDied","Data":"d9f4717b45e9bdecc417f8fd8b5ad7425621ac342483367d52d67c62922ffd7c"} Nov 02 22:14:26.405999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.393545 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:26.606710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.596173 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d47f8605-swxqm\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:26.606710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.596289 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d47f8605-swxqm\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:26.606710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.596529 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnsqk\" (UniqueName: \"kubernetes.io/projected/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-kube-api-access-pnsqk\") pod \"maas-api-revoke-keys-e2e-ait-d47f8605-swxqm\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:26.607071 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.606914 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:14:26.629699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.629525 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 02 22:14:26.740612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.738649 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d47f8605-swxqm\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:26.740612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.738753 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d47f8605-swxqm\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:26.740612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.738883 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnsqk\" (UniqueName: \"kubernetes.io/projected/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-kube-api-access-pnsqk\") pod \"maas-api-revoke-keys-e2e-ait-d47f8605-swxqm\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:26.745725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.741884 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d47f8605-swxqm\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:26.779660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.779142 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:26.779660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.779553 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-585h6" podUID="3768a351-a0c8-4472-8769-d39331d91f5a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:26.852708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.851701 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:26.862107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.859413 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:26.862107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.860768 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:14:26.862107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.861062 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:14:26.862107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:26.861302 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:14:27.046520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.046452 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnsqk\" (UniqueName: \"kubernetes.io/projected/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-kube-api-access-pnsqk\") pod \"maas-api-revoke-keys-e2e-ait-d47f8605-swxqm\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:27.151582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.151503 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d47f8605-swxqm\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:27.203486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.198783 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Nov 02 22:14:27.203814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.198825 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:27.287150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.285564 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:14:27.287150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.285936 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:14:27.287150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.286188 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:14:27.287150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.286435 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:14:27.681427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.680816 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:27.819726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.809929 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:14:27.889697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.887194 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:14:27.914869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.914831 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:27.915457 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.915421 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:14:27.915936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.915903 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:14:27.916359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:27.916333 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:14:28.302700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.292790 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:28.302700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.293295 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:14:28.302700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.293873 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:28.302700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.294437 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:14:28.363725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.360085 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:28.391439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.391349 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:14:28.391732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.391713 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:14:28.500996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.500411 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:14:28.583620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.579636 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 02 22:14:28.583620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.582602 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-xks-ca\") pod \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " Nov 02 22:14:28.583620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.582749 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-openshift-ca\") pod \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " Nov 02 22:14:28.583620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.582953 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-kube-api-access-zb6zt\" (UniqueName: \"kubernetes.io/projected/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-kube-api-access-zb6zt\") pod \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\" (UID: \"2276b6bc-c7f2-43d1-ae6b-87f04c73c197\") " Nov 02 22:14:28.626777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.624922 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-openshift-ca" pod "2276b6bc-c7f2-43d1-ae6b-87f04c73c197" (UID: "2276b6bc-c7f2-43d1-ae6b-87f04c73c197"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:28.660750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.636027 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:28.703796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.703753 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:28.704640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.704613 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:14:28.824838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.824794 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:28.888535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.888503 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-kube-api-access-zb6zt" pod "2276b6bc-c7f2-43d1-ae6b-87f04c73c197" (UID: "2276b6bc-c7f2-43d1-ae6b-87f04c73c197"). InnerVolumeSpecName "kube-api-access-zb6zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:28.894378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.894336 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-xks-ca" pod "2276b6bc-c7f2-43d1-ae6b-87f04c73c197" (UID: "2276b6bc-c7f2-43d1-ae6b-87f04c73c197"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:28.950317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.950283 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:14:28.970991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.968973 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:28.970991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.969005 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb6zt\" (UniqueName: \"kubernetes.io/projected/2276b6bc-c7f2-43d1-ae6b-87f04c73c197-kube-api-access-zb6zt\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:28.999578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.999442 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:28.999779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:28.999756 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:14:29.000868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.000045 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:14:29.015486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.015137 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm"] Nov 02 22:14:29.019769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.019696 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:14:29.019769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.019746 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:14:29.037494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.037468 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:14:29.108984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.108955 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:14:29.115087 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.114845 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:14:29.117784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.117767 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:14:29.118301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.118199 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:14:29.118301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.118221 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:14:29.277539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.277453 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" Nov 02 22:14:29.277539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.277477 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq" event={"ID":"2276b6bc-c7f2-43d1-ae6b-87f04c73c197","Type":"ContainerDied","Data":"0998da4ebd6df1d268ae1f73705a7d6d29141005607175fa9185834bdcff0ef5"} Nov 02 22:14:29.277539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.277534 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0998da4ebd6df1d268ae1f73705a7d6d29141005607175fa9185834bdcff0ef5" Nov 02 22:14:29.279012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:29.278987 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" event={"ID":"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe","Type":"ContainerStarted","Data":"db2e79d0400b05880d52fe476e53d4515219582a4d87ab214ffaaa02ec3b81ca"} Nov 02 22:14:30.292990 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.292956 4309 generic.go:356] "Generic (PLEG): container finished" podID="f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe" containerID="c49131e60c4d863e6023011fda02e324a8137247a19f9aa59fbaf66776a91313" exitCode=0 Nov 02 22:14:30.293460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.293433 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" event={"ID":"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe","Type":"ContainerDied","Data":"c49131e60c4d863e6023011fda02e324a8137247a19f9aa59fbaf66776a91313"} Nov 02 22:14:30.344793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.344760 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn"] Nov 02 22:14:30.346150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.346121 4309 cpu_manager.go:414] "containerMap: removing container" podUID="2276b6bc-c7f2-43d1-ae6b-87f04c73c197" containerName="revoke-keys" Nov 02 22:14:30.346150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.346147 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2276b6bc-c7f2-43d1-ae6b-87f04c73c197" containerName="revoke-keys" Nov 02 22:14:30.346372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.346356 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="2276b6bc-c7f2-43d1-ae6b-87f04c73c197" containerName="revoke-keys" Nov 02 22:14:30.358259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.358230 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.361521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.361496 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn"] Nov 02 22:14:30.470958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.470882 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtc6j\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-kube-api-access-dtc6j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.470958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.470919 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.471137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.471047 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee5122-b354-4878-ad0b-bed64f19aad9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.471137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.471117 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.471209 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.471193 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.471245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.471219 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.471317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.471304 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.573347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.573312 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtc6j\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-kube-api-access-dtc6j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.573518 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.573361 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.573619 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.573604 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee5122-b354-4878-ad0b-bed64f19aad9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.573820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.573799 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.573929 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.573897 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.573929 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.573922 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.574042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.574027 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.575022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.574993 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.576911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.576802 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.577403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.577372 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.577649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.577588 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee5122-b354-4878-ad0b-bed64f19aad9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.577649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.577599 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.581786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.581762 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.581786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.581771 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtc6j\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-kube-api-access-dtc6j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-n6rpn\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.675037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.675006 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:30.986397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.986364 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn"] Nov 02 22:14:30.989199 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:14:30.989166 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ee5122_b354_4878_ad0b_bed64f19aad9.slice/crio-260c5bed38323edac2f271f7d8b09de4ef4d28b7c3fc8301beb2a5fbdb918666 WatchSource:0}: Error finding container 260c5bed38323edac2f271f7d8b09de4ef4d28b7c3fc8301beb2a5fbdb918666: Status 404 returned error can't find the container with id 260c5bed38323edac2f271f7d8b09de4ef4d28b7c3fc8301beb2a5fbdb918666 Nov 02 22:14:30.993216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.993113 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:14:30.993216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:30.993202 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:14:31.318409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.318372 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" event={"ID":"60ee5122-b354-4878-ad0b-bed64f19aad9","Type":"ContainerStarted","Data":"1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f"} Nov 02 22:14:31.318982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.318448 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:31.318982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.318466 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" event={"ID":"60ee5122-b354-4878-ad0b-bed64f19aad9","Type":"ContainerStarted","Data":"260c5bed38323edac2f271f7d8b09de4ef4d28b7c3fc8301beb2a5fbdb918666"} Nov 02 22:14:31.339241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.339190 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" podStartSLOduration=1.339176304 podStartE2EDuration="1.339176304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:31.336645763 +0000 UTC m=+999.274758981" watchObservedRunningTime="2026-11-02 22:14:31.339176304 +0000 UTC m=+999.277289521" Nov 02 22:14:31.839780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.839748 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:31.980128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.980035 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-openshift-ca\") pod \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " Nov 02 22:14:31.980128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.980102 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-kube-api-access-pnsqk\" (UniqueName: \"kubernetes.io/projected/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-kube-api-access-pnsqk\") pod \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " Nov 02 22:14:31.980370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.980140 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-xks-ca\") pod \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\" (UID: \"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe\") " Nov 02 22:14:31.980921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.980891 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-openshift-ca" pod "f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe" (UID: "f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:31.984645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.984479 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-kube-api-access-pnsqk" pod "f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe" (UID: "f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe"). InnerVolumeSpecName "kube-api-access-pnsqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:31.994848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:31.994804 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-xks-ca" pod "f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe" (UID: "f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:32.086000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:32.085331 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:32.086495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:32.086425 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnsqk\" (UniqueName: \"kubernetes.io/projected/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-kube-api-access-pnsqk\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:32.086657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:32.086645 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:32.361256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:32.361217 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" event={"ID":"f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe","Type":"ContainerDied","Data":"db2e79d0400b05880d52fe476e53d4515219582a4d87ab214ffaaa02ec3b81ca"} Nov 02 22:14:32.361821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:32.361238 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm" Nov 02 22:14:32.362128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:32.362107 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db2e79d0400b05880d52fe476e53d4515219582a4d87ab214ffaaa02ec3b81ca" Nov 02 22:14:32.364204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:32.364178 4309 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-n6rpn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:14:32.364318 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:32.364250 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" podUID="60ee5122-b354-4878-ad0b-bed64f19aad9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:14:35.363460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:35.363426 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:14:36.306795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.306755 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk"] Nov 02 22:14:36.307590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.307570 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe" containerName="revoke-keys" Nov 02 22:14:36.307728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.307592 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe" containerName="revoke-keys" Nov 02 22:14:36.307875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.307860 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe" containerName="revoke-keys" Nov 02 22:14:36.317169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.317145 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.317822 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.317799 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk"] Nov 02 22:14:36.320378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.320357 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:36.329956 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.329932 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:14:36.330085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.330023 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:36.330085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.330054 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:36.445200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.445163 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.445574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.445260 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.445574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.445290 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdg9s\" (UniqueName: \"kubernetes.io/projected/05a6559a-2962-4867-ad2c-32eba11d2beb-kube-api-access-wdg9s\") pod \"maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.554741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.551121 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdg9s\" (UniqueName: \"kubernetes.io/projected/05a6559a-2962-4867-ad2c-32eba11d2beb-kube-api-access-wdg9s\") pod \"maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.554741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.551551 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.554741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.551617 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.554741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.552415 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.555337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.555317 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.560518 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.560492 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdg9s\" (UniqueName: \"kubernetes.io/projected/05a6559a-2962-4867-ad2c-32eba11d2beb-kube-api-access-wdg9s\") pod \"maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:36.650153 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:36.650118 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:37.145550 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:14:37.145521 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a6559a_2962_4867_ad2c_32eba11d2beb.slice/crio-2728039a5ef49982d6a4960abeda9aa340a1f5042dbcd9216ea3b8bc88e79dfa WatchSource:0}: Error finding container 2728039a5ef49982d6a4960abeda9aa340a1f5042dbcd9216ea3b8bc88e79dfa: Status 404 returned error can't find the container with id 2728039a5ef49982d6a4960abeda9aa340a1f5042dbcd9216ea3b8bc88e79dfa Nov 02 22:14:37.148529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:37.148499 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk"] Nov 02 22:14:37.418334 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:37.418304 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" event={"ID":"05a6559a-2962-4867-ad2c-32eba11d2beb","Type":"ContainerStarted","Data":"2728039a5ef49982d6a4960abeda9aa340a1f5042dbcd9216ea3b8bc88e79dfa"} Nov 02 22:14:38.434509 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:38.434430 4309 generic.go:356] "Generic (PLEG): container finished" podID="05a6559a-2962-4867-ad2c-32eba11d2beb" containerID="1ef47db951c0e6978b7a877b27f371be263ae3f64ad2e5ad4625977d4d8c7c26" exitCode=0 Nov 02 22:14:38.435045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:38.434507 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" event={"ID":"05a6559a-2962-4867-ad2c-32eba11d2beb","Type":"ContainerDied","Data":"1ef47db951c0e6978b7a877b27f371be263ae3f64ad2e5ad4625977d4d8c7c26"} Nov 02 22:14:39.644767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.644740 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:39.707650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.707565 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a6559a-2962-4867-ad2c-32eba11d2beb-kube-api-access-wdg9s\" (UniqueName: \"kubernetes.io/projected/05a6559a-2962-4867-ad2c-32eba11d2beb-kube-api-access-wdg9s\") pod \"05a6559a-2962-4867-ad2c-32eba11d2beb\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " Nov 02 22:14:39.707845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.707804 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-xks-ca\") pod \"05a6559a-2962-4867-ad2c-32eba11d2beb\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " Nov 02 22:14:39.708171 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.708145 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-openshift-ca\") pod \"05a6559a-2962-4867-ad2c-32eba11d2beb\" (UID: \"05a6559a-2962-4867-ad2c-32eba11d2beb\") " Nov 02 22:14:39.708601 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.708570 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-openshift-ca" pod "05a6559a-2962-4867-ad2c-32eba11d2beb" (UID: "05a6559a-2962-4867-ad2c-32eba11d2beb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:39.709152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.709126 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:39.710724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.710188 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a6559a-2962-4867-ad2c-32eba11d2beb-kube-api-access-wdg9s" pod "05a6559a-2962-4867-ad2c-32eba11d2beb" (UID: "05a6559a-2962-4867-ad2c-32eba11d2beb"). InnerVolumeSpecName "kube-api-access-wdg9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:39.710724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.710518 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-xks-ca" pod "05a6559a-2962-4867-ad2c-32eba11d2beb" (UID: "05a6559a-2962-4867-ad2c-32eba11d2beb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:39.810855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.810820 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdg9s\" (UniqueName: \"kubernetes.io/projected/05a6559a-2962-4867-ad2c-32eba11d2beb-kube-api-access-wdg9s\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:39.810855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:39.810853 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a6559a-2962-4867-ad2c-32eba11d2beb-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:40.458796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:40.458655 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" Nov 02 22:14:40.459098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:40.458694 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk" event={"ID":"05a6559a-2962-4867-ad2c-32eba11d2beb","Type":"ContainerDied","Data":"2728039a5ef49982d6a4960abeda9aa340a1f5042dbcd9216ea3b8bc88e79dfa"} Nov 02 22:14:40.459206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:40.459127 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2728039a5ef49982d6a4960abeda9aa340a1f5042dbcd9216ea3b8bc88e79dfa" Nov 02 22:14:41.631220 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:14:41.631169 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:14:42.840502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:42.840466 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq"] Nov 02 22:14:42.849366 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:42.848905 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8da3b70c-btfhq"] Nov 02 22:14:43.634310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.634270 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v"] Nov 02 22:14:43.635503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.635482 4309 cpu_manager.go:414] "containerMap: removing container" podUID="05a6559a-2962-4867-ad2c-32eba11d2beb" containerName="revoke-keys" Nov 02 22:14:43.635503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.635502 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a6559a-2962-4867-ad2c-32eba11d2beb" containerName="revoke-keys" Nov 02 22:14:43.635661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.635651 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a6559a-2962-4867-ad2c-32eba11d2beb" containerName="revoke-keys" Nov 02 22:14:43.639963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.639941 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.642423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.642393 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v"] Nov 02 22:14:43.644856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.644834 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:43.645117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.645098 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:43.645208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.645139 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:43.645277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.645208 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:14:43.728456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.728422 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb9dk\" (UniqueName: \"kubernetes.io/projected/f7ddf629-6246-4035-b2d6-219e83a1a5b3-kube-api-access-pb9dk\") pod \"maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.728623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.728498 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.728774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.728755 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.831200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.831162 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.831417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.831398 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb9dk\" (UniqueName: \"kubernetes.io/projected/f7ddf629-6246-4035-b2d6-219e83a1a5b3-kube-api-access-pb9dk\") pod \"maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.831484 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.831469 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.832131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.832105 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.841111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.841079 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.843105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.843083 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb9dk\" (UniqueName: \"kubernetes.io/projected/f7ddf629-6246-4035-b2d6-219e83a1a5b3-kube-api-access-pb9dk\") pod \"maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:43.960186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:43.960096 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:44.256758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:44.237495 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2276b6bc-c7f2-43d1-ae6b-87f04c73c197" path="/var/lib/kubelet/pods/2276b6bc-c7f2-43d1-ae6b-87f04c73c197/volumes" Nov 02 22:14:44.547216 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:14:44.547182 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ddf629_6246_4035_b2d6_219e83a1a5b3.slice/crio-520c1febd9cb3880a77d1e280bfffad2b35520d91dc715e1b52d082120dcbb58 WatchSource:0}: Error finding container 520c1febd9cb3880a77d1e280bfffad2b35520d91dc715e1b52d082120dcbb58: Status 404 returned error can't find the container with id 520c1febd9cb3880a77d1e280bfffad2b35520d91dc715e1b52d082120dcbb58 Nov 02 22:14:44.550313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:44.550287 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v"] Nov 02 22:14:44.590565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:44.590523 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" event={"ID":"f7ddf629-6246-4035-b2d6-219e83a1a5b3","Type":"ContainerStarted","Data":"520c1febd9cb3880a77d1e280bfffad2b35520d91dc715e1b52d082120dcbb58"} Nov 02 22:14:45.059492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:45.059342 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm"] Nov 02 22:14:45.069705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:45.069661 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d47f8605-swxqm"] Nov 02 22:14:45.613642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:45.613453 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" event={"ID":"f7ddf629-6246-4035-b2d6-219e83a1a5b3","Type":"ContainerStarted","Data":"ff04574973b59e5ee210410ad769c2b7eda9b62e879c4b2b856fa1334297a94d"} Nov 02 22:14:46.193181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:46.193144 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe" path="/var/lib/kubelet/pods/f7c6017a-1ec2-4f61-80a5-572f4eaf2ffe/volumes" Nov 02 22:14:46.482621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:46.482522 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n"] Nov 02 22:14:46.635931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:46.635893 4309 generic.go:356] "Generic (PLEG): container finished" podID="f7ddf629-6246-4035-b2d6-219e83a1a5b3" containerID="ff04574973b59e5ee210410ad769c2b7eda9b62e879c4b2b856fa1334297a94d" exitCode=0 Nov 02 22:14:46.636120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:46.635982 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" event={"ID":"f7ddf629-6246-4035-b2d6-219e83a1a5b3","Type":"ContainerDied","Data":"ff04574973b59e5ee210410ad769c2b7eda9b62e879c4b2b856fa1334297a94d"} Nov 02 22:14:47.019267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.019238 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:47.080251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.080210 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-xks-ca\") pod \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " Nov 02 22:14:47.080488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.080465 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ddf629-6246-4035-b2d6-219e83a1a5b3-kube-api-access-pb9dk\" (UniqueName: \"kubernetes.io/projected/f7ddf629-6246-4035-b2d6-219e83a1a5b3-kube-api-access-pb9dk\") pod \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " Nov 02 22:14:47.080539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.080520 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-openshift-ca\") pod \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\" (UID: \"f7ddf629-6246-4035-b2d6-219e83a1a5b3\") " Nov 02 22:14:47.081908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.081879 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-openshift-ca" pod "f7ddf629-6246-4035-b2d6-219e83a1a5b3" (UID: "f7ddf629-6246-4035-b2d6-219e83a1a5b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:47.085116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.085081 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ddf629-6246-4035-b2d6-219e83a1a5b3-kube-api-access-pb9dk" pod "f7ddf629-6246-4035-b2d6-219e83a1a5b3" (UID: "f7ddf629-6246-4035-b2d6-219e83a1a5b3"). InnerVolumeSpecName "kube-api-access-pb9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:47.090651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.090621 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-xks-ca" pod "f7ddf629-6246-4035-b2d6-219e83a1a5b3" (UID: "f7ddf629-6246-4035-b2d6-219e83a1a5b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:14:47.184502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.184438 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:47.185453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.184472 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb9dk\" (UniqueName: \"kubernetes.io/projected/f7ddf629-6246-4035-b2d6-219e83a1a5b3-kube-api-access-pb9dk\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:47.185717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.185665 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7ddf629-6246-4035-b2d6-219e83a1a5b3-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:47.784275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.784221 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" event={"ID":"f7ddf629-6246-4035-b2d6-219e83a1a5b3","Type":"ContainerDied","Data":"520c1febd9cb3880a77d1e280bfffad2b35520d91dc715e1b52d082120dcbb58"} Nov 02 22:14:47.784743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.784291 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="520c1febd9cb3880a77d1e280bfffad2b35520d91dc715e1b52d082120dcbb58" Nov 02 22:14:47.784743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:47.784369 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v" Nov 02 22:14:50.094732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:50.092452 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk"] Nov 02 22:14:50.104010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:50.103975 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a72dd63-fp8wk"] Nov 02 22:14:50.193118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:50.193082 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a6559a-2962-4867-ad2c-32eba11d2beb" path="/var/lib/kubelet/pods/05a6559a-2962-4867-ad2c-32eba11d2beb/volumes" Nov 02 22:14:51.483041 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:51.483004 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" podUID="27f299ed-7957-4ade-9951-36e5f02ecc0c" containerName="payload-processing" containerID="cri-o://44cdc84e59b9e2d37ec1e26ef2fd8d8215866f6ff4ef709058bd16401654dd13" gracePeriod=25 Nov 02 22:14:52.030429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:52.030330 4309 generic.go:356] "Generic (PLEG): container finished" podID="27f299ed-7957-4ade-9951-36e5f02ecc0c" containerID="44cdc84e59b9e2d37ec1e26ef2fd8d8215866f6ff4ef709058bd16401654dd13" exitCode=0 Nov 02 22:14:52.030868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:52.030399 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" event={"ID":"27f299ed-7957-4ade-9951-36e5f02ecc0c","Type":"ContainerDied","Data":"44cdc84e59b9e2d37ec1e26ef2fd8d8215866f6ff4ef709058bd16401654dd13"} Nov 02 22:14:52.053942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:52.053913 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:52.059950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:52.059922 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f299ed-7957-4ade-9951-36e5f02ecc0c-kube-api-access-ltzpn\" (UniqueName: \"kubernetes.io/projected/27f299ed-7957-4ade-9951-36e5f02ecc0c-kube-api-access-ltzpn\") pod \"27f299ed-7957-4ade-9951-36e5f02ecc0c\" (UID: \"27f299ed-7957-4ade-9951-36e5f02ecc0c\") " Nov 02 22:14:52.060135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:52.060021 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f299ed-7957-4ade-9951-36e5f02ecc0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f299ed-7957-4ade-9951-36e5f02ecc0c-plugins-config-volume\") pod \"27f299ed-7957-4ade-9951-36e5f02ecc0c\" (UID: \"27f299ed-7957-4ade-9951-36e5f02ecc0c\") " Nov 02 22:14:52.060398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:52.060368 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f299ed-7957-4ade-9951-36e5f02ecc0c-plugins-config-volume" pod "27f299ed-7957-4ade-9951-36e5f02ecc0c" (UID: "27f299ed-7957-4ade-9951-36e5f02ecc0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:14:52.060783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:52.060766 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27f299ed-7957-4ade-9951-36e5f02ecc0c-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:52.062834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:52.062806 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f299ed-7957-4ade-9951-36e5f02ecc0c-kube-api-access-ltzpn" pod "27f299ed-7957-4ade-9951-36e5f02ecc0c" (UID: "27f299ed-7957-4ade-9951-36e5f02ecc0c"). InnerVolumeSpecName "kube-api-access-ltzpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:14:52.162026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:52.161999 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzpn\" (UniqueName: \"kubernetes.io/projected/27f299ed-7957-4ade-9951-36e5f02ecc0c-kube-api-access-ltzpn\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:14:52.296733 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:14:52.296637 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:14:53.123659 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:53.123622 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" event={"ID":"27f299ed-7957-4ade-9951-36e5f02ecc0c","Type":"ContainerDied","Data":"8d09600f5aec392fb8606072b8bd708c1b17047b17001b843f21f11993b02ac5"} Nov 02 22:14:53.124391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:53.124369 4309 scope.go:122] "RemoveContainer" containerID="44cdc84e59b9e2d37ec1e26ef2fd8d8215866f6ff4ef709058bd16401654dd13" Nov 02 22:14:53.124790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:53.124571 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n" Nov 02 22:14:53.235012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:53.233827 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n"] Nov 02 22:14:53.277430 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:53.277375 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a72dd63-864f4ffd9-7w55n"] Nov 02 22:14:54.206714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:54.206637 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f299ed-7957-4ade-9951-36e5f02ecc0c" path="/var/lib/kubelet/pods/27f299ed-7957-4ade-9951-36e5f02ecc0c/volumes" Nov 02 22:14:55.668197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.663954 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5"] Nov 02 22:14:55.668197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.664770 4309 cpu_manager.go:414] "containerMap: removing container" podUID="27f299ed-7957-4ade-9951-36e5f02ecc0c" containerName="payload-processing" Nov 02 22:14:55.668197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.664786 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f299ed-7957-4ade-9951-36e5f02ecc0c" containerName="payload-processing" Nov 02 22:14:55.668197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.664828 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f7ddf629-6246-4035-b2d6-219e83a1a5b3" containerName="revoke-keys" Nov 02 22:14:55.668197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.664834 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ddf629-6246-4035-b2d6-219e83a1a5b3" containerName="revoke-keys" Nov 02 22:14:55.668197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.665022 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ddf629-6246-4035-b2d6-219e83a1a5b3" containerName="revoke-keys" Nov 02 22:14:55.668197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.665040 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f299ed-7957-4ade-9951-36e5f02ecc0c" containerName="payload-processing" Nov 02 22:14:55.685546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.675368 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:55.723698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.714080 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:14:55.723698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.714333 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5"] Nov 02 22:14:55.723698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.715306 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:14:55.723698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.715415 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:14:55.723698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.715479 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:14:55.832317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.832269 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mzl\" (UniqueName: \"kubernetes.io/projected/af95b18e-259a-4658-b937-f26fe81fbd20-kube-api-access-c9mzl\") pod \"maas-api-revoke-sub-f1c75b42-vszf5\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:55.832502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.832479 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1c75b42-vszf5\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:55.832593 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.832578 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1c75b42-vszf5\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:55.938471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.936380 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1c75b42-vszf5\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:55.938471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.936492 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1c75b42-vszf5\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:55.938471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.936770 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mzl\" (UniqueName: \"kubernetes.io/projected/af95b18e-259a-4658-b937-f26fe81fbd20-kube-api-access-c9mzl\") pod \"maas-api-revoke-sub-f1c75b42-vszf5\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:55.938471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:55.938294 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1c75b42-vszf5\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:56.066382 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:56.053181 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1c75b42-vszf5\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:56.101701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:56.089763 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mzl\" (UniqueName: \"kubernetes.io/projected/af95b18e-259a-4658-b937-f26fe81fbd20-kube-api-access-c9mzl\") pod \"maas-api-revoke-sub-f1c75b42-vszf5\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:56.277529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:56.268995 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v"] Nov 02 22:14:56.346947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:56.342028 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:14:56.480040 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:56.471152 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb16f94c-2cm7v"] Nov 02 22:14:56.753534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:56.753473 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp"] Nov 02 22:14:56.754432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:56.754394 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" podUID="d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" containerName="istio-proxy" containerID="cri-o://c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444" gracePeriod=30 Nov 02 22:14:57.980729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:57.978885 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5"] Nov 02 22:14:58.242748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:58.242038 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ddf629-6246-4035-b2d6-219e83a1a5b3" path="/var/lib/kubelet/pods/f7ddf629-6246-4035-b2d6-219e83a1a5b3/volumes" Nov 02 22:14:58.946803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:58.946694 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" event={"ID":"af95b18e-259a-4658-b937-f26fe81fbd20","Type":"ContainerStarted","Data":"974b811ceffb2156bbb0db411d641cadff72f65ed1e1f2631fe3cc3d18f472eb"} Nov 02 22:14:58.946803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:58.946766 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" event={"ID":"af95b18e-259a-4658-b937-f26fe81fbd20","Type":"ContainerStarted","Data":"4686aaefe95d638b76cf81cd8d011b10c724b618629f9de122068c59dec78efc"} Nov 02 22:14:58.964591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:14:58.964534 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" podStartSLOduration=3.964516218 podStartE2EDuration="3.964516218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:14:58.962866294 +0000 UTC m=+1026.900979513" watchObservedRunningTime="2026-11-02 22:14:58.964516218 +0000 UTC m=+1026.902629440" Nov 02 22:15:00.064697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.061492 4309 generic.go:356] "Generic (PLEG): container finished" podID="af95b18e-259a-4658-b937-f26fe81fbd20" containerID="974b811ceffb2156bbb0db411d641cadff72f65ed1e1f2631fe3cc3d18f472eb" exitCode=0 Nov 02 22:15:00.064697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.062012 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" event={"ID":"af95b18e-259a-4658-b937-f26fe81fbd20","Type":"ContainerDied","Data":"974b811ceffb2156bbb0db411d641cadff72f65ed1e1f2631fe3cc3d18f472eb"} Nov 02 22:15:00.167736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.163275 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48"] Nov 02 22:15:00.169971 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.169945 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:00.264876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.263280 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48\" (UID: \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:00.264876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.263435 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdqs\" (UniqueName: \"kubernetes.io/projected/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-kube-api-access-8hdqs\") pod \"maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48\" (UID: \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:00.277865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.277839 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw"] Nov 02 22:15:00.298904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.298872 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7"] Nov 02 22:15:00.306890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.306862 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:00.374204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.368424 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48\" (UID: \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:00.374204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.368529 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdqs\" (UniqueName: \"kubernetes.io/projected/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-kube-api-access-8hdqs\") pod \"maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48\" (UID: \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:00.374204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.369507 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48\" (UID: \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:00.378550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.378530 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd"] Nov 02 22:15:00.379148 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.379127 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:00.380013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.379794 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdqs\" (UniqueName: \"kubernetes.io/projected/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-kube-api-access-8hdqs\") pod \"maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48\" (UID: \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:00.459252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.459221 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48"] Nov 02 22:15:00.459529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.459512 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw"] Nov 02 22:15:00.459777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.459759 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:00.465924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.465857 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7"] Nov 02 22:15:00.472182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.470502 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4vj\" (UniqueName: \"kubernetes.io/projected/91cb5416-01d5-45ba-8481-44f634c81348-kube-api-access-rb4vj\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7\" (UID: \"91cb5416-01d5-45ba-8481-44f634c81348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:00.472182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.470689 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw\" (UID: \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:00.472182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.470742 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cb5416-01d5-45ba-8481-44f634c81348-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7\" (UID: \"91cb5416-01d5-45ba-8481-44f634c81348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:00.472182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.470807 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktc7l\" (UniqueName: \"kubernetes.io/projected/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-kube-api-access-ktc7l\") pod \"maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw\" (UID: \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.573996 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.577081 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd"] Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.621256 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pv5t\" (UniqueName: \"kubernetes.io/projected/6bb8fb52-562c-4c1b-88a8-e1000eba3678-kube-api-access-2pv5t\") pod \"maas-api-key-cleanup-29894295-gvbcd\" (UID: \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.621426 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw\" (UID: \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.621472 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cb5416-01d5-45ba-8481-44f634c81348-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7\" (UID: \"91cb5416-01d5-45ba-8481-44f634c81348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.621534 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktc7l\" (UniqueName: \"kubernetes.io/projected/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-kube-api-access-ktc7l\") pod \"maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw\" (UID: \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.621746 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb8fb52-562c-4c1b-88a8-e1000eba3678-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-gvbcd\" (UID: \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.621863 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4vj\" (UniqueName: \"kubernetes.io/projected/91cb5416-01d5-45ba-8481-44f634c81348-kube-api-access-rb4vj\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7\" (UID: \"91cb5416-01d5-45ba-8481-44f634c81348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.646985 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw\" (UID: \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.647713 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cb5416-01d5-45ba-8481-44f634c81348-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7\" (UID: \"91cb5416-01d5-45ba-8481-44f634c81348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:00.702905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.693346 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktc7l\" (UniqueName: \"kubernetes.io/projected/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-kube-api-access-ktc7l\") pod \"maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw\" (UID: \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:00.731818 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.729946 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4"] Nov 02 22:15:00.733783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.733746 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb8fb52-562c-4c1b-88a8-e1000eba3678-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-gvbcd\" (UID: \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:00.734165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.734129 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pv5t\" (UniqueName: \"kubernetes.io/projected/6bb8fb52-562c-4c1b-88a8-e1000eba3678-kube-api-access-2pv5t\") pod \"maas-api-key-cleanup-29894295-gvbcd\" (UID: \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:00.742699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.738837 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb8fb52-562c-4c1b-88a8-e1000eba3678-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894295-gvbcd\" (UID: \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:00.765399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.765365 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:00.776288 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.771853 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:00.812729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.811614 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4vj\" (UniqueName: \"kubernetes.io/projected/91cb5416-01d5-45ba-8481-44f634c81348-kube-api-access-rb4vj\") pod \"maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7\" (UID: \"91cb5416-01d5-45ba-8481-44f634c81348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:00.855470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.855425 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pv5t\" (UniqueName: \"kubernetes.io/projected/6bb8fb52-562c-4c1b-88a8-e1000eba3678-kube-api-access-2pv5t\") pod \"maas-api-key-cleanup-29894295-gvbcd\" (UID: \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:00.942876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.942847 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6"] Nov 02 22:15:00.957431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.956522 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:00.958089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.958062 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c871e58-223f-425c-8874-63c90a1ec02d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4\" (UID: \"8c871e58-223f-425c-8874-63c90a1ec02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:00.958770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.958726 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qbg\" (UniqueName: \"kubernetes.io/projected/8c871e58-223f-425c-8874-63c90a1ec02d-kube-api-access-76qbg\") pod \"maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4\" (UID: \"8c871e58-223f-425c-8874-63c90a1ec02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:00.979830 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:00.979802 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4"] Nov 02 22:15:01.057700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.056734 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6"] Nov 02 22:15:01.061513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.061481 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hptz\" (UniqueName: \"kubernetes.io/projected/90135336-acb5-401c-88ce-457247da8afb-kube-api-access-5hptz\") pod \"maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6\" (UID: \"90135336-acb5-401c-88ce-457247da8afb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:01.061947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.061917 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qbg\" (UniqueName: \"kubernetes.io/projected/8c871e58-223f-425c-8874-63c90a1ec02d-kube-api-access-76qbg\") pod \"maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4\" (UID: \"8c871e58-223f-425c-8874-63c90a1ec02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:01.101189 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.101156 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c871e58-223f-425c-8874-63c90a1ec02d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4\" (UID: \"8c871e58-223f-425c-8874-63c90a1ec02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:01.101894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.101865 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90135336-acb5-401c-88ce-457247da8afb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6\" (UID: \"90135336-acb5-401c-88ce-457247da8afb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:01.121606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.099775 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qbg\" (UniqueName: \"kubernetes.io/projected/8c871e58-223f-425c-8874-63c90a1ec02d-kube-api-access-76qbg\") pod \"maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4\" (UID: \"8c871e58-223f-425c-8874-63c90a1ec02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:01.122720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.122578 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c871e58-223f-425c-8874-63c90a1ec02d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4\" (UID: \"8c871e58-223f-425c-8874-63c90a1ec02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:01.122878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.077976 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:01.123726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.082932 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:01.203959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.203928 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hptz\" (UniqueName: \"kubernetes.io/projected/90135336-acb5-401c-88ce-457247da8afb-kube-api-access-5hptz\") pod \"maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6\" (UID: \"90135336-acb5-401c-88ce-457247da8afb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:01.204397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.204376 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90135336-acb5-401c-88ce-457247da8afb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6\" (UID: \"90135336-acb5-401c-88ce-457247da8afb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:01.205782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.205617 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90135336-acb5-401c-88ce-457247da8afb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6\" (UID: \"90135336-acb5-401c-88ce-457247da8afb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:01.350404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.306485 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hptz\" (UniqueName: \"kubernetes.io/projected/90135336-acb5-401c-88ce-457247da8afb-kube-api-access-5hptz\") pod \"maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6\" (UID: \"90135336-acb5-401c-88ce-457247da8afb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:01.370712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.356901 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:01.373966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:01.372037 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:02.247550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.247520 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48"] Nov 02 22:15:02.436101 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.436070 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw"] Nov 02 22:15:02.588629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.588576 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd"] Nov 02 22:15:02.692492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.688017 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7"] Nov 02 22:15:02.740164 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:15:02.740114 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cb5416_01d5_45ba_8481_44f634c81348.slice/crio-04e480d53d317ccdc803d63937bc9026cd3e93c47413375bd11443e142897d80 WatchSource:0}: Error finding container 04e480d53d317ccdc803d63937bc9026cd3e93c47413375bd11443e142897d80: Status 404 returned error can't find the container with id 04e480d53d317ccdc803d63937bc9026cd3e93c47413375bd11443e142897d80 Nov 02 22:15:02.745310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.745284 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:15:02.840726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.840136 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4"] Nov 02 22:15:02.841496 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:02.841164 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:02.848638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.847262 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-xks-ca\") pod \"af95b18e-259a-4658-b937-f26fe81fbd20\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " Nov 02 22:15:02.848638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.847407 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af95b18e-259a-4658-b937-f26fe81fbd20-kube-api-access-c9mzl\" (UniqueName: \"kubernetes.io/projected/af95b18e-259a-4658-b937-f26fe81fbd20-kube-api-access-c9mzl\") pod \"af95b18e-259a-4658-b937-f26fe81fbd20\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " Nov 02 22:15:02.848638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.847638 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-openshift-ca\") pod \"af95b18e-259a-4658-b937-f26fe81fbd20\" (UID: \"af95b18e-259a-4658-b937-f26fe81fbd20\") " Nov 02 22:15:02.849614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.849282 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-openshift-ca" pod "af95b18e-259a-4658-b937-f26fe81fbd20" (UID: "af95b18e-259a-4658-b937-f26fe81fbd20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:02.856198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.856177 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af95b18e-259a-4658-b937-f26fe81fbd20-kube-api-access-c9mzl" pod "af95b18e-259a-4658-b937-f26fe81fbd20" (UID: "af95b18e-259a-4658-b937-f26fe81fbd20"). InnerVolumeSpecName "kube-api-access-c9mzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:02.860718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.860631 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-xks-ca" pod "af95b18e-259a-4658-b937-f26fe81fbd20" (UID: "af95b18e-259a-4658-b937-f26fe81fbd20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:15:02.950320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.950279 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9mzl\" (UniqueName: \"kubernetes.io/projected/af95b18e-259a-4658-b937-f26fe81fbd20-kube-api-access-c9mzl\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:02.950320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.950316 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:02.950520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.950330 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af95b18e-259a-4658-b937-f26fe81fbd20-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:02.951568 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:15:02.951541 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90135336_acb5_401c_88ce_457247da8afb.slice/crio-66ffd386e13e1af5a063dac1f4a5c3c34092e1ad95ecc370679dc9980346b484 WatchSource:0}: Error finding container 66ffd386e13e1af5a063dac1f4a5c3c34092e1ad95ecc370679dc9980346b484: Status 404 returned error can't find the container with id 66ffd386e13e1af5a063dac1f4a5c3c34092e1ad95ecc370679dc9980346b484 Nov 02 22:15:02.956754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:02.956648 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6"] Nov 02 22:15:03.192338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.192306 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" event={"ID":"8c871e58-223f-425c-8874-63c90a1ec02d","Type":"ContainerStarted","Data":"c03639678d087c17dc158bdb6afad243396ffd0da1a83678fca8ecc78b967aa6"} Nov 02 22:15:03.194413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.194385 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" Nov 02 22:15:03.194706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.194667 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5" event={"ID":"af95b18e-259a-4658-b937-f26fe81fbd20","Type":"ContainerDied","Data":"4686aaefe95d638b76cf81cd8d011b10c724b618629f9de122068c59dec78efc"} Nov 02 22:15:03.194817 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.194735 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4686aaefe95d638b76cf81cd8d011b10c724b618629f9de122068c59dec78efc" Nov 02 22:15:03.198938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.198876 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" event={"ID":"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5","Type":"ContainerStarted","Data":"a3437adbcb8e4edab44023a9dacb06d11eb1c72557705fc5365237cfa3f0d09e"} Nov 02 22:15:03.200510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.200482 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" event={"ID":"91cb5416-01d5-45ba-8481-44f634c81348","Type":"ContainerStarted","Data":"04e480d53d317ccdc803d63937bc9026cd3e93c47413375bd11443e142897d80"} Nov 02 22:15:03.202714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.202691 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" event={"ID":"6bb8fb52-562c-4c1b-88a8-e1000eba3678","Type":"ContainerStarted","Data":"dbc8c9692e3f54f5b20371a351a2bbd1b950f28d490826f1b0f6ce9c06872748"} Nov 02 22:15:03.206134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.205523 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" event={"ID":"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff","Type":"ContainerStarted","Data":"b9f606e413b9b3beb20da433aa7daa02d0723a17d647104d5fee8e9fae33ed3d"} Nov 02 22:15:03.206338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.206311 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" event={"ID":"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff","Type":"ContainerStarted","Data":"b9e807540b7be75b5427ee2ba7078906e981294665e5ec52b923e3ab16d19f28"} Nov 02 22:15:03.207410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.207355 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" event={"ID":"90135336-acb5-401c-88ce-457247da8afb","Type":"ContainerStarted","Data":"66ffd386e13e1af5a063dac1f4a5c3c34092e1ad95ecc370679dc9980346b484"} Nov 02 22:15:03.233588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:03.233512 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" podStartSLOduration=3.233490454 podStartE2EDuration="3.233490454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:03.224219092 +0000 UTC m=+1031.162332310" watchObservedRunningTime="2026-11-02 22:15:03.233490454 +0000 UTC m=+1031.171603674" Nov 02 22:15:04.219664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.219622 4309 generic.go:356] "Generic (PLEG): container finished" podID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerID="3d52d2ec6eb3e62a0e9bc86929b6c6849b4d79e574f2fa3cda889598bd016d36" exitCode=60 Nov 02 22:15:04.220090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.219719 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" event={"ID":"6bb8fb52-562c-4c1b-88a8-e1000eba3678","Type":"ContainerDied","Data":"3d52d2ec6eb3e62a0e9bc86929b6c6849b4d79e574f2fa3cda889598bd016d36"} Nov 02 22:15:04.220090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.220081 4309 scope.go:122] "RemoveContainer" containerID="3d52d2ec6eb3e62a0e9bc86929b6c6849b4d79e574f2fa3cda889598bd016d36" Nov 02 22:15:04.222383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.222309 4309 generic.go:356] "Generic (PLEG): container finished" podID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerID="b9f606e413b9b3beb20da433aa7daa02d0723a17d647104d5fee8e9fae33ed3d" exitCode=60 Nov 02 22:15:04.222489 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.222352 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" event={"ID":"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff","Type":"ContainerDied","Data":"b9f606e413b9b3beb20da433aa7daa02d0723a17d647104d5fee8e9fae33ed3d"} Nov 02 22:15:04.222662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.222645 4309 scope.go:122] "RemoveContainer" containerID="b9f606e413b9b3beb20da433aa7daa02d0723a17d647104d5fee8e9fae33ed3d" Nov 02 22:15:04.230536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.230511 4309 generic.go:356] "Generic (PLEG): container finished" podID="90135336-acb5-401c-88ce-457247da8afb" containerID="70e9aed035ba18724ca31927a890c29bb07294ebb591e3874ef81749d8a60d1c" exitCode=60 Nov 02 22:15:04.230702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.230661 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" event={"ID":"90135336-acb5-401c-88ce-457247da8afb","Type":"ContainerDied","Data":"70e9aed035ba18724ca31927a890c29bb07294ebb591e3874ef81749d8a60d1c"} Nov 02 22:15:04.230897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.230880 4309 scope.go:122] "RemoveContainer" containerID="70e9aed035ba18724ca31927a890c29bb07294ebb591e3874ef81749d8a60d1c" Nov 02 22:15:04.237340 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.237314 4309 generic.go:356] "Generic (PLEG): container finished" podID="8c871e58-223f-425c-8874-63c90a1ec02d" containerID="33a9a03734e0f69e65a27660047bc48abb3d965cd5eabf0a97d0d4febdac6cbc" exitCode=60 Nov 02 22:15:04.237526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.237508 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" event={"ID":"8c871e58-223f-425c-8874-63c90a1ec02d","Type":"ContainerDied","Data":"33a9a03734e0f69e65a27660047bc48abb3d965cd5eabf0a97d0d4febdac6cbc"} Nov 02 22:15:04.238018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.237995 4309 scope.go:122] "RemoveContainer" containerID="33a9a03734e0f69e65a27660047bc48abb3d965cd5eabf0a97d0d4febdac6cbc" Nov 02 22:15:04.245028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.244502 4309 generic.go:356] "Generic (PLEG): container finished" podID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerID="4f2069bca2b53c2092519eab9144ee97bb50810a9799129ca44d405a7017bc67" exitCode=60 Nov 02 22:15:04.245028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.244606 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" event={"ID":"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5","Type":"ContainerDied","Data":"4f2069bca2b53c2092519eab9144ee97bb50810a9799129ca44d405a7017bc67"} Nov 02 22:15:04.245028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.244929 4309 scope.go:122] "RemoveContainer" containerID="4f2069bca2b53c2092519eab9144ee97bb50810a9799129ca44d405a7017bc67" Nov 02 22:15:04.258618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.258590 4309 generic.go:356] "Generic (PLEG): container finished" podID="91cb5416-01d5-45ba-8481-44f634c81348" containerID="58adb63dddbd448fc9589d955b4d9866fca4669153a83d91fe1333472f329652" exitCode=60 Nov 02 22:15:04.258864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.258843 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" event={"ID":"91cb5416-01d5-45ba-8481-44f634c81348","Type":"ContainerDied","Data":"58adb63dddbd448fc9589d955b4d9866fca4669153a83d91fe1333472f329652"} Nov 02 22:15:04.259955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:04.259935 4309 scope.go:122] "RemoveContainer" containerID="58adb63dddbd448fc9589d955b4d9866fca4669153a83d91fe1333472f329652" Nov 02 22:15:04.275710 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:04.275660 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:04.289805 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:04.289774 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:05.273515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.273342 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" event={"ID":"90135336-acb5-401c-88ce-457247da8afb","Type":"ContainerStarted","Data":"52e877a29dd46f9a347861fceffde802feafffb3a921717d264e4e11e5b14626"} Nov 02 22:15:05.275934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.275902 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" event={"ID":"8c871e58-223f-425c-8874-63c90a1ec02d","Type":"ContainerStarted","Data":"7327580c68e320df9a81f0dbf6d8145acf9b24764e0f5becbd1a26534683081b"} Nov 02 22:15:05.278349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.278321 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" event={"ID":"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5","Type":"ContainerStarted","Data":"7a0b654626154060a4a40992e57912cd07d855ac203babeda3498d126747833c"} Nov 02 22:15:05.280661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.280640 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" event={"ID":"91cb5416-01d5-45ba-8481-44f634c81348","Type":"ContainerStarted","Data":"1bbb8a91f2e24a7a5e60249fa8379e35c5aa29966f476533d8ee5189286cce0d"} Nov 02 22:15:05.306739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.291600 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" podStartSLOduration=5.291584181 podStartE2EDuration="5.291584181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:05.290321268 +0000 UTC m=+1033.228434485" watchObservedRunningTime="2026-11-02 22:15:05.291584181 +0000 UTC m=+1033.229697400" Nov 02 22:15:05.306739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.293499 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" event={"ID":"6bb8fb52-562c-4c1b-88a8-e1000eba3678","Type":"ContainerStarted","Data":"07741e13b453ca8415ff7678ec93f53edbdcfe20adfa83e2e21ac328662ea090"} Nov 02 22:15:05.317474 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.317437 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" event={"ID":"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff","Type":"ContainerStarted","Data":"2373f259acf360cbf4d7c5ed5169de6c15f9f90dbb62b0a756c65ae9261dd18f"} Nov 02 22:15:05.349785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.349722 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" podStartSLOduration=5.349701772 podStartE2EDuration="5.349701772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:05.336934012 +0000 UTC m=+1033.275047234" watchObservedRunningTime="2026-11-02 22:15:05.349701772 +0000 UTC m=+1033.287814992" Nov 02 22:15:05.369548 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.369338 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" podStartSLOduration=5.369320606 podStartE2EDuration="5.369320606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:05.351910197 +0000 UTC m=+1033.290023417" watchObservedRunningTime="2026-11-02 22:15:05.369320606 +0000 UTC m=+1033.307433826" Nov 02 22:15:05.381015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.380954 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" podStartSLOduration=5.380916821 podStartE2EDuration="5.380916821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:05.36713336 +0000 UTC m=+1033.305246577" watchObservedRunningTime="2026-11-02 22:15:05.380916821 +0000 UTC m=+1033.319030040" Nov 02 22:15:05.404063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:05.403999 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" podStartSLOduration=5.40397899 podStartE2EDuration="5.40397899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:15:05.386130833 +0000 UTC m=+1033.324244053" watchObservedRunningTime="2026-11-02 22:15:05.40397899 +0000 UTC m=+1033.342092209" Nov 02 22:15:05.890053 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:05.889106 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:06.387360 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.386758 4309 generic.go:356] "Generic (PLEG): container finished" podID="90135336-acb5-401c-88ce-457247da8afb" containerID="52e877a29dd46f9a347861fceffde802feafffb3a921717d264e4e11e5b14626" exitCode=60 Nov 02 22:15:06.387360 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.387012 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" event={"ID":"90135336-acb5-401c-88ce-457247da8afb","Type":"ContainerDied","Data":"52e877a29dd46f9a347861fceffde802feafffb3a921717d264e4e11e5b14626"} Nov 02 22:15:06.387360 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.387102 4309 scope.go:122] "RemoveContainer" containerID="70e9aed035ba18724ca31927a890c29bb07294ebb591e3874ef81749d8a60d1c" Nov 02 22:15:06.401224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.387697 4309 scope.go:122] "RemoveContainer" containerID="52e877a29dd46f9a347861fceffde802feafffb3a921717d264e4e11e5b14626" Nov 02 22:15:06.401224 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:06.388176 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6_odh-ai-gateway-infra(90135336-acb5-401c-88ce-457247da8afb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" podUID="90135336-acb5-401c-88ce-457247da8afb" Nov 02 22:15:06.401737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.401462 4309 generic.go:356] "Generic (PLEG): container finished" podID="8c871e58-223f-425c-8874-63c90a1ec02d" containerID="7327580c68e320df9a81f0dbf6d8145acf9b24764e0f5becbd1a26534683081b" exitCode=60 Nov 02 22:15:06.401737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.401698 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" event={"ID":"8c871e58-223f-425c-8874-63c90a1ec02d","Type":"ContainerDied","Data":"7327580c68e320df9a81f0dbf6d8145acf9b24764e0f5becbd1a26534683081b"} Nov 02 22:15:06.403774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.402205 4309 scope.go:122] "RemoveContainer" containerID="7327580c68e320df9a81f0dbf6d8145acf9b24764e0f5becbd1a26534683081b" Nov 02 22:15:06.403774 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:06.402719 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4_odh-ai-gateway-infra(8c871e58-223f-425c-8874-63c90a1ec02d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" Nov 02 22:15:06.406499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.406018 4309 generic.go:356] "Generic (PLEG): container finished" podID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerID="7a0b654626154060a4a40992e57912cd07d855ac203babeda3498d126747833c" exitCode=60 Nov 02 22:15:06.406499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.406117 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" event={"ID":"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5","Type":"ContainerDied","Data":"7a0b654626154060a4a40992e57912cd07d855ac203babeda3498d126747833c"} Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.413543 4309 generic.go:356] "Generic (PLEG): container finished" podID="91cb5416-01d5-45ba-8481-44f634c81348" containerID="1bbb8a91f2e24a7a5e60249fa8379e35c5aa29966f476533d8ee5189286cce0d" exitCode=60 Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.413814 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" event={"ID":"91cb5416-01d5-45ba-8481-44f634c81348","Type":"ContainerDied","Data":"1bbb8a91f2e24a7a5e60249fa8379e35c5aa29966f476533d8ee5189286cce0d"} Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.422877 4309 scope.go:122] "RemoveContainer" containerID="7a0b654626154060a4a40992e57912cd07d855ac203babeda3498d126747833c" Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.422943 4309 generic.go:356] "Generic (PLEG): container finished" podID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerID="07741e13b453ca8415ff7678ec93f53edbdcfe20adfa83e2e21ac328662ea090" exitCode=60 Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.423025 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" event={"ID":"6bb8fb52-562c-4c1b-88a8-e1000eba3678","Type":"ContainerDied","Data":"07741e13b453ca8415ff7678ec93f53edbdcfe20adfa83e2e21ac328662ea090"} Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:06.423340 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw_odh-ai-gateway-infra(e9f78b57-bc54-4aee-a235-0e4f0da0ffa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.424751 4309 scope.go:122] "RemoveContainer" containerID="1bbb8a91f2e24a7a5e60249fa8379e35c5aa29966f476533d8ee5189286cce0d" Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.424948 4309 scope.go:122] "RemoveContainer" containerID="07741e13b453ca8415ff7678ec93f53edbdcfe20adfa83e2e21ac328662ea090" Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:06.425241 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7_odh-ai-gateway-infra(91cb5416-01d5-45ba-8481-44f634c81348)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" podUID="91cb5416-01d5-45ba-8481-44f634c81348" Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:06.426058 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-gvbcd_odh-ai-gateway-infra(6bb8fb52-562c-4c1b-88a8-e1000eba3678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.428190 4309 generic.go:356] "Generic (PLEG): container finished" podID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerID="2373f259acf360cbf4d7c5ed5169de6c15f9f90dbb62b0a756c65ae9261dd18f" exitCode=60 Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.428272 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" event={"ID":"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff","Type":"ContainerDied","Data":"2373f259acf360cbf4d7c5ed5169de6c15f9f90dbb62b0a756c65ae9261dd18f"} Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.428725 4309 scope.go:122] "RemoveContainer" containerID="2373f259acf360cbf4d7c5ed5169de6c15f9f90dbb62b0a756c65ae9261dd18f" Nov 02 22:15:06.449703 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:06.429031 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48_odh-ai-gateway-infra(8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" Nov 02 22:15:06.489042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.481324 4309 scope.go:122] "RemoveContainer" containerID="33a9a03734e0f69e65a27660047bc48abb3d965cd5eabf0a97d0d4febdac6cbc" Nov 02 22:15:06.513233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.513204 4309 scope.go:122] "RemoveContainer" containerID="4f2069bca2b53c2092519eab9144ee97bb50810a9799129ca44d405a7017bc67" Nov 02 22:15:06.536821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.536788 4309 scope.go:122] "RemoveContainer" containerID="58adb63dddbd448fc9589d955b4d9866fca4669153a83d91fe1333472f329652" Nov 02 22:15:06.559162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.559140 4309 scope.go:122] "RemoveContainer" containerID="3d52d2ec6eb3e62a0e9bc86929b6c6849b4d79e574f2fa3cda889598bd016d36" Nov 02 22:15:06.596031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:06.596004 4309 scope.go:122] "RemoveContainer" containerID="b9f606e413b9b3beb20da433aa7daa02d0723a17d647104d5fee8e9fae33ed3d" Nov 02 22:15:07.446169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:07.446139 4309 scope.go:122] "RemoveContainer" containerID="07741e13b453ca8415ff7678ec93f53edbdcfe20adfa83e2e21ac328662ea090" Nov 02 22:15:07.446714 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:07.446614 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-gvbcd_odh-ai-gateway-infra(6bb8fb52-562c-4c1b-88a8-e1000eba3678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" Nov 02 22:15:07.448332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:07.448313 4309 scope.go:122] "RemoveContainer" containerID="2373f259acf360cbf4d7c5ed5169de6c15f9f90dbb62b0a756c65ae9261dd18f" Nov 02 22:15:07.448795 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:07.448761 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48_odh-ai-gateway-infra(8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" Nov 02 22:15:07.450571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:07.450549 4309 scope.go:122] "RemoveContainer" containerID="52e877a29dd46f9a347861fceffde802feafffb3a921717d264e4e11e5b14626" Nov 02 22:15:07.451004 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:07.450970 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6_odh-ai-gateway-infra(90135336-acb5-401c-88ce-457247da8afb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" podUID="90135336-acb5-401c-88ce-457247da8afb" Nov 02 22:15:07.452426 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:07.452396 4309 scope.go:122] "RemoveContainer" containerID="7327580c68e320df9a81f0dbf6d8145acf9b24764e0f5becbd1a26534683081b" Nov 02 22:15:07.453718 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:07.453642 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4_odh-ai-gateway-infra(8c871e58-223f-425c-8874-63c90a1ec02d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" Nov 02 22:15:07.456420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:07.456398 4309 scope.go:122] "RemoveContainer" containerID="7a0b654626154060a4a40992e57912cd07d855ac203babeda3498d126747833c" Nov 02 22:15:07.456810 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:07.456782 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw_odh-ai-gateway-infra(e9f78b57-bc54-4aee-a235-0e4f0da0ffa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" Nov 02 22:15:07.458876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:07.458854 4309 scope.go:122] "RemoveContainer" containerID="1bbb8a91f2e24a7a5e60249fa8379e35c5aa29966f476533d8ee5189286cce0d" Nov 02 22:15:07.459208 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:07.459186 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7_odh-ai-gateway-infra(91cb5416-01d5-45ba-8481-44f634c81348)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" podUID="91cb5416-01d5-45ba-8481-44f634c81348" Nov 02 22:15:09.575963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:09.561077 4309 patch_prober.go:29] interesting pod/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": read tcp 10.134.0.2:58740->10.134.0.137:15021: read: connection reset by peer" start-of-body= Nov 02 22:15:09.575963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:09.561139 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" podUID="d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": read tcp 10.134.0.2:58740->10.134.0.137:15021: read: connection reset by peer" Nov 02 22:15:10.308794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.307654 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:15:10.456624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.456524 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-socket\") pod \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " Nov 02 22:15:10.456854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.456631 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-kube-api-access-hjqmc\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-kube-api-access-hjqmc\") pod \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " Nov 02 22:15:10.456854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.456709 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-podinfo\") pod \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " Nov 02 22:15:10.457033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.456851 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-envoy\") pod \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " Nov 02 22:15:10.457033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.456896 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istiod-ca-cert\") pod \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " Nov 02 22:15:10.457141 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.457074 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-data\") pod \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " Nov 02 22:15:10.457215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.457143 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-credential-socket\") pod \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " Nov 02 22:15:10.457269 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.457211 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-certs\") pod \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " Nov 02 22:15:10.457323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.457281 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-token\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-token\") pod \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\" (UID: \"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19\") " Nov 02 22:15:10.458800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.458751 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-data" pod "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" (UID: "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:10.459032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.458925 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istiod-ca-cert" pod "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" (UID: "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:10.459128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.458909 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-credential-socket" pod "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" (UID: "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:10.459128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.459010 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-certs" pod "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" (UID: "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:10.459128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.459111 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-socket" pod "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" (UID: "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:10.462923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.461398 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-token" pod "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" (UID: "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:10.462923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.461620 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-kube-api-access-hjqmc" pod "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" (UID: "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19"). InnerVolumeSpecName "kube-api-access-hjqmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:10.466278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.465952 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-podinfo" pod "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" (UID: "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:15:10.466278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.466027 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-envoy" pod "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" (UID: "d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:15:10.504641 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.504603 4309 generic.go:356] "Generic (PLEG): container finished" podID="d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" containerID="c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444" exitCode=0 Nov 02 22:15:10.504833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.504663 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" Nov 02 22:15:10.504833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.504716 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" event={"ID":"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19","Type":"ContainerDied","Data":"c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444"} Nov 02 22:15:10.504833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.504774 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp" event={"ID":"d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19","Type":"ContainerDied","Data":"fd082032f8b8c9da069caf9362eb91a25d1e38a076b81e376a9e061390ee1a81"} Nov 02 22:15:10.504833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.504796 4309 scope.go:122] "RemoveContainer" containerID="c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444" Nov 02 22:15:10.507299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.507275 4309 generic.go:356] "Generic (PLEG): container finished" podID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerID="51189d2b30d4cd50e2c713d1a0f4a1749e3201feed477b5ebeca4e3690e69142" exitCode=0 Nov 02 22:15:10.507439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.507354 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" event={"ID":"2972de31-06c1-4022-a3ab-3796c5ecd136","Type":"ContainerDied","Data":"51189d2b30d4cd50e2c713d1a0f4a1749e3201feed477b5ebeca4e3690e69142"} Nov 02 22:15:10.507856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.507838 4309 scope.go:122] "RemoveContainer" containerID="51189d2b30d4cd50e2c713d1a0f4a1749e3201feed477b5ebeca4e3690e69142" Nov 02 22:15:10.510473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.510441 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:15:10.510561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.510529 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:15:10.510688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.510577 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:15:10.519584 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.519564 4309 scope.go:122] "RemoveContainer" containerID="c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444" Nov 02 22:15:10.519961 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:10.519936 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444\": container with ID starting with c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444 not found: ID does not exist" containerID="c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444" Nov 02 22:15:10.520052 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.519971 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444"} err="failed to get container status \"c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444\": rpc error: code = NotFound desc = could not find container \"c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444\": container with ID starting with c0afb9d5b3a21b3b0885461d1e083bf19eb5cbc00eff02cf8d864bd4c2d45444 not found: ID does not exist" Nov 02 22:15:10.560724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.560491 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:10.560724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.560613 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:10.560724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.560633 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjqmc\" (UniqueName: \"kubernetes.io/projected/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-kube-api-access-hjqmc\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:10.560724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.560649 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:10.560724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.560664 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:10.560724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.560695 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:10.560724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.560710 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:10.560724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.560723 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:10.561258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.560735 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:10.561875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.561854 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp"] Nov 02 22:15:10.578055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:10.577999 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bb16f94c-gw-openshift-default-7757d5f58f-94kzp"] Nov 02 22:15:11.522429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:11.522382 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" event={"ID":"2972de31-06c1-4022-a3ab-3796c5ecd136","Type":"ContainerStarted","Data":"8fd577ea6d5363cc23a0285127405b58b6bf01ac2b31deeb154844b8c0ef89ac"} Nov 02 22:15:11.608304 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:11.608261 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:15:12.045056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:12.045023 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:15:12.059706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:12.052731 4309 patch_prober.go:29] interesting pod/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:15:12.059706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:12.052811 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 22:15:12.256635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:12.256590 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" path="/var/lib/kubelet/pods/d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19/volumes" Nov 02 22:15:13.049623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:13.049153 4309 patch_prober.go:29] interesting pod/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:15:13.049623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:13.049229 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 22:15:13.073127 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:13.073100 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:14.043889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:14.043852 4309 patch_prober.go:29] interesting pod/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:15:14.044075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:14.043920 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 22:15:14.317863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:14.317821 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:15:15.043521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:15.043489 4309 patch_prober.go:29] interesting pod/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:15:15.043722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:15.043556 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 22:15:16.042978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:16.042945 4309 patch_prober.go:29] interesting pod/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:15:16.043462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:16.043013 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 22:15:16.183714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:16.182116 4309 scope.go:122] "RemoveContainer" containerID="7a0b654626154060a4a40992e57912cd07d855ac203babeda3498d126747833c" Nov 02 22:15:16.862326 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:16.862280 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" event={"ID":"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5","Type":"ContainerStarted","Data":"1d4882699b43190cc6e55d836ee3b10d9a78e0e7cc4431bbd54b74d4af0623e3"} Nov 02 22:15:17.043602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.043515 4309 patch_prober.go:29] interesting pod/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Nov 02 22:15:17.043602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.043578 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Nov 02 22:15:17.181523 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.181496 4309 scope.go:122] "RemoveContainer" containerID="7327580c68e320df9a81f0dbf6d8145acf9b24764e0f5becbd1a26534683081b" Nov 02 22:15:17.181758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.181697 4309 scope.go:122] "RemoveContainer" containerID="07741e13b453ca8415ff7678ec93f53edbdcfe20adfa83e2e21ac328662ea090" Nov 02 22:15:17.879127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.879019 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" event={"ID":"6bb8fb52-562c-4c1b-88a8-e1000eba3678","Type":"ContainerStarted","Data":"41930a99022283c8fd38980ba16edbd873a554eddb908730b1c91276a8ef9cc6"} Nov 02 22:15:17.887782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.882263 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" event={"ID":"8c871e58-223f-425c-8874-63c90a1ec02d","Type":"ContainerStarted","Data":"f0544fa0ab8c37fe7c94fb4820514ee85d5fd99322a39ba871dfa7e5fe1b3128"} Nov 02 22:15:17.887782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.884966 4309 generic.go:356] "Generic (PLEG): container finished" podID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerID="1d4882699b43190cc6e55d836ee3b10d9a78e0e7cc4431bbd54b74d4af0623e3" exitCode=60 Nov 02 22:15:17.887782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.885007 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" event={"ID":"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5","Type":"ContainerDied","Data":"1d4882699b43190cc6e55d836ee3b10d9a78e0e7cc4431bbd54b74d4af0623e3"} Nov 02 22:15:17.887782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.885080 4309 scope.go:122] "RemoveContainer" containerID="7a0b654626154060a4a40992e57912cd07d855ac203babeda3498d126747833c" Nov 02 22:15:17.887782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.885336 4309 scope.go:122] "RemoveContainer" containerID="1d4882699b43190cc6e55d836ee3b10d9a78e0e7cc4431bbd54b74d4af0623e3" Nov 02 22:15:17.887782 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:17.885653 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw_odh-ai-gateway-infra(e9f78b57-bc54-4aee-a235-0e4f0da0ffa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" Nov 02 22:15:17.929780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:17.929512 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw"] Nov 02 22:15:18.047936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.047908 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:15:18.049134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.049106 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:15:18.181339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.181264 4309 scope.go:122] "RemoveContainer" containerID="1bbb8a91f2e24a7a5e60249fa8379e35c5aa29966f476533d8ee5189286cce0d" Nov 02 22:15:18.927133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.925930 4309 generic.go:356] "Generic (PLEG): container finished" podID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerID="41930a99022283c8fd38980ba16edbd873a554eddb908730b1c91276a8ef9cc6" exitCode=60 Nov 02 22:15:18.927133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.926241 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" event={"ID":"6bb8fb52-562c-4c1b-88a8-e1000eba3678","Type":"ContainerDied","Data":"41930a99022283c8fd38980ba16edbd873a554eddb908730b1c91276a8ef9cc6"} Nov 02 22:15:18.927133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.926326 4309 scope.go:122] "RemoveContainer" containerID="07741e13b453ca8415ff7678ec93f53edbdcfe20adfa83e2e21ac328662ea090" Nov 02 22:15:18.929974 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.927658 4309 scope.go:122] "RemoveContainer" containerID="41930a99022283c8fd38980ba16edbd873a554eddb908730b1c91276a8ef9cc6" Nov 02 22:15:18.929974 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:18.928168 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894295-gvbcd_odh-ai-gateway-infra(6bb8fb52-562c-4c1b-88a8-e1000eba3678)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" Nov 02 22:15:18.939445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.939415 4309 generic.go:356] "Generic (PLEG): container finished" podID="8c871e58-223f-425c-8874-63c90a1ec02d" containerID="f0544fa0ab8c37fe7c94fb4820514ee85d5fd99322a39ba871dfa7e5fe1b3128" exitCode=60 Nov 02 22:15:18.939895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.939869 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" event={"ID":"8c871e58-223f-425c-8874-63c90a1ec02d","Type":"ContainerDied","Data":"f0544fa0ab8c37fe7c94fb4820514ee85d5fd99322a39ba871dfa7e5fe1b3128"} Nov 02 22:15:18.941515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.941493 4309 scope.go:122] "RemoveContainer" containerID="f0544fa0ab8c37fe7c94fb4820514ee85d5fd99322a39ba871dfa7e5fe1b3128" Nov 02 22:15:18.942106 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:18.942086 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4_odh-ai-gateway-infra(8c871e58-223f-425c-8874-63c90a1ec02d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" Nov 02 22:15:18.946935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.946913 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd"] Nov 02 22:15:18.957358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.957064 4309 scope.go:122] "RemoveContainer" containerID="7327580c68e320df9a81f0dbf6d8145acf9b24764e0f5becbd1a26534683081b" Nov 02 22:15:18.958168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.958142 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" event={"ID":"91cb5416-01d5-45ba-8481-44f634c81348","Type":"ContainerStarted","Data":"e94b940d732342ef07c0812c1aa048a9955c375f479689d3a919b3ce370e08f1"} Nov 02 22:15:18.969467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:18.969404 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4"] Nov 02 22:15:19.176381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.176356 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:19.181257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.181193 4309 scope.go:122] "RemoveContainer" containerID="2373f259acf360cbf4d7c5ed5169de6c15f9f90dbb62b0a756c65ae9261dd18f" Nov 02 22:15:19.316914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.315417 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-kube-api-access-ktc7l\" (UniqueName: \"kubernetes.io/projected/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-kube-api-access-ktc7l\") pod \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\" (UID: \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\") " Nov 02 22:15:19.316914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.315625 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-maas-api-service-ca\") pod \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\" (UID: \"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5\") " Nov 02 22:15:19.318416 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.318384 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-maas-api-service-ca" pod "e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" (UID: "e9f78b57-bc54-4aee-a235-0e4f0da0ffa5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:19.334471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.334439 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-kube-api-access-ktc7l" pod "e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" (UID: "e9f78b57-bc54-4aee-a235-0e4f0da0ffa5"). InnerVolumeSpecName "kube-api-access-ktc7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:19.420367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.420328 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktc7l\" (UniqueName: \"kubernetes.io/projected/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-kube-api-access-ktc7l\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.420552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.420418 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5-maas-api-service-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:19.979799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.979770 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" Nov 02 22:15:19.979964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.979761 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw" event={"ID":"e9f78b57-bc54-4aee-a235-0e4f0da0ffa5","Type":"ContainerDied","Data":"a3437adbcb8e4edab44023a9dacb06d11eb1c72557705fc5365237cfa3f0d09e"} Nov 02 22:15:19.981277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.981245 4309 scope.go:122] "RemoveContainer" containerID="1d4882699b43190cc6e55d836ee3b10d9a78e0e7cc4431bbd54b74d4af0623e3" Nov 02 22:15:19.982645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.982617 4309 generic.go:356] "Generic (PLEG): container finished" podID="91cb5416-01d5-45ba-8481-44f634c81348" containerID="e94b940d732342ef07c0812c1aa048a9955c375f479689d3a919b3ce370e08f1" exitCode=60 Nov 02 22:15:19.982862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.982842 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" event={"ID":"91cb5416-01d5-45ba-8481-44f634c81348","Type":"ContainerDied","Data":"e94b940d732342ef07c0812c1aa048a9955c375f479689d3a919b3ce370e08f1"} Nov 02 22:15:19.983396 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.983337 4309 scope.go:122] "RemoveContainer" containerID="e94b940d732342ef07c0812c1aa048a9955c375f479689d3a919b3ce370e08f1" Nov 02 22:15:19.986843 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:19.986737 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7_odh-ai-gateway-infra(91cb5416-01d5-45ba-8481-44f634c81348)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" podUID="91cb5416-01d5-45ba-8481-44f634c81348" Nov 02 22:15:19.996077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:19.995150 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" event={"ID":"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff","Type":"ContainerStarted","Data":"3c481d4f66668cff991642f3da90b6505b0be02f8186bcb625cd8680376c13aa"} Nov 02 22:15:20.009771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.008222 4309 scope.go:122] "RemoveContainer" containerID="1bbb8a91f2e24a7a5e60249fa8379e35c5aa29966f476533d8ee5189286cce0d" Nov 02 22:15:20.031315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.025105 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw"] Nov 02 22:15:20.061914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.061869 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2532af-29894295-slbnw"] Nov 02 22:15:20.067451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.067426 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7"] Nov 02 22:15:20.188837 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.188803 4309 scope.go:122] "RemoveContainer" containerID="52e877a29dd46f9a347861fceffde802feafffb3a921717d264e4e11e5b14626" Nov 02 22:15:20.217797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.215802 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" path="/var/lib/kubelet/pods/e9f78b57-bc54-4aee-a235-0e4f0da0ffa5/volumes" Nov 02 22:15:20.335926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.335891 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:20.373767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.373734 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c871e58-223f-425c-8874-63c90a1ec02d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c871e58-223f-425c-8874-63c90a1ec02d-maas-api-service-ca\") pod \"8c871e58-223f-425c-8874-63c90a1ec02d\" (UID: \"8c871e58-223f-425c-8874-63c90a1ec02d\") " Nov 02 22:15:20.373920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.373888 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c871e58-223f-425c-8874-63c90a1ec02d-kube-api-access-76qbg\" (UniqueName: \"kubernetes.io/projected/8c871e58-223f-425c-8874-63c90a1ec02d-kube-api-access-76qbg\") pod \"8c871e58-223f-425c-8874-63c90a1ec02d\" (UID: \"8c871e58-223f-425c-8874-63c90a1ec02d\") " Nov 02 22:15:20.374324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.374265 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c871e58-223f-425c-8874-63c90a1ec02d-maas-api-service-ca" pod "8c871e58-223f-425c-8874-63c90a1ec02d" (UID: "8c871e58-223f-425c-8874-63c90a1ec02d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:20.376739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.376713 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c871e58-223f-425c-8874-63c90a1ec02d-kube-api-access-76qbg" pod "8c871e58-223f-425c-8874-63c90a1ec02d" (UID: "8c871e58-223f-425c-8874-63c90a1ec02d"). InnerVolumeSpecName "kube-api-access-76qbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:20.376902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.376884 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76qbg\" (UniqueName: \"kubernetes.io/projected/8c871e58-223f-425c-8874-63c90a1ec02d-kube-api-access-76qbg\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.376983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.376908 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c871e58-223f-425c-8874-63c90a1ec02d-maas-api-service-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.379352 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.378913 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:20.478287 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.478247 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb8fb52-562c-4c1b-88a8-e1000eba3678-kube-api-access-2pv5t\" (UniqueName: \"kubernetes.io/projected/6bb8fb52-562c-4c1b-88a8-e1000eba3678-kube-api-access-2pv5t\") pod \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\" (UID: \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\") " Nov 02 22:15:20.478480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.478384 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb8fb52-562c-4c1b-88a8-e1000eba3678-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb8fb52-562c-4c1b-88a8-e1000eba3678-maas-api-service-ca\") pod \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\" (UID: \"6bb8fb52-562c-4c1b-88a8-e1000eba3678\") " Nov 02 22:15:20.478754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.478729 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb8fb52-562c-4c1b-88a8-e1000eba3678-maas-api-service-ca" pod "6bb8fb52-562c-4c1b-88a8-e1000eba3678" (UID: "6bb8fb52-562c-4c1b-88a8-e1000eba3678"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:20.479166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.479140 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bb8fb52-562c-4c1b-88a8-e1000eba3678-maas-api-service-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:20.483368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.483343 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb8fb52-562c-4c1b-88a8-e1000eba3678-kube-api-access-2pv5t" pod "6bb8fb52-562c-4c1b-88a8-e1000eba3678" (UID: "6bb8fb52-562c-4c1b-88a8-e1000eba3678"). InnerVolumeSpecName "kube-api-access-2pv5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:20.580730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:20.580635 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pv5t\" (UniqueName: \"kubernetes.io/projected/6bb8fb52-562c-4c1b-88a8-e1000eba3678-kube-api-access-2pv5t\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:21.006984 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:21.006872 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:21.017967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.015405 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" event={"ID":"6bb8fb52-562c-4c1b-88a8-e1000eba3678","Type":"ContainerDied","Data":"dbc8c9692e3f54f5b20371a351a2bbd1b950f28d490826f1b0f6ce9c06872748"} Nov 02 22:15:21.017967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.015439 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd" Nov 02 22:15:21.017967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.015485 4309 scope.go:122] "RemoveContainer" containerID="41930a99022283c8fd38980ba16edbd873a554eddb908730b1c91276a8ef9cc6" Nov 02 22:15:21.019756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.019061 4309 generic.go:356] "Generic (PLEG): container finished" podID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerID="3c481d4f66668cff991642f3da90b6505b0be02f8186bcb625cd8680376c13aa" exitCode=60 Nov 02 22:15:21.019756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.019185 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" event={"ID":"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff","Type":"ContainerDied","Data":"3c481d4f66668cff991642f3da90b6505b0be02f8186bcb625cd8680376c13aa"} Nov 02 22:15:21.019756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.019709 4309 scope.go:122] "RemoveContainer" containerID="3c481d4f66668cff991642f3da90b6505b0be02f8186bcb625cd8680376c13aa" Nov 02 22:15:21.025425 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:21.020212 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48_odh-ai-gateway-infra(8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" Nov 02 22:15:21.025425 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.022942 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" event={"ID":"90135336-acb5-401c-88ce-457247da8afb","Type":"ContainerStarted","Data":"62887ce05f089fd17fa5eb08c0fde4ca8276cb11787a036e0ba689aa2f298baa"} Nov 02 22:15:21.027855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.026628 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" event={"ID":"8c871e58-223f-425c-8874-63c90a1ec02d","Type":"ContainerDied","Data":"c03639678d087c17dc158bdb6afad243396ffd0da1a83678fca8ecc78b967aa6"} Nov 02 22:15:21.027855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.026661 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4" Nov 02 22:15:21.048218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.048195 4309 scope.go:122] "RemoveContainer" containerID="2373f259acf360cbf4d7c5ed5169de6c15f9f90dbb62b0a756c65ae9261dd18f" Nov 02 22:15:21.069864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.069840 4309 scope.go:122] "RemoveContainer" containerID="f0544fa0ab8c37fe7c94fb4820514ee85d5fd99322a39ba871dfa7e5fe1b3128" Nov 02 22:15:21.073888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.073861 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd"] Nov 02 22:15:21.086903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.086851 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894295-gvbcd"] Nov 02 22:15:21.105936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.105906 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4"] Nov 02 22:15:21.115436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.115405 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-361a44-29894295-jknn4"] Nov 02 22:15:21.138143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.138117 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48"] Nov 02 22:15:21.262835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.255667 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:21.317746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.315909 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91cb5416-01d5-45ba-8481-44f634c81348-kube-api-access-rb4vj\" (UniqueName: \"kubernetes.io/projected/91cb5416-01d5-45ba-8481-44f634c81348-kube-api-access-rb4vj\") pod \"91cb5416-01d5-45ba-8481-44f634c81348\" (UID: \"91cb5416-01d5-45ba-8481-44f634c81348\") " Nov 02 22:15:21.318695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.318654 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91cb5416-01d5-45ba-8481-44f634c81348-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cb5416-01d5-45ba-8481-44f634c81348-maas-api-service-ca\") pod \"91cb5416-01d5-45ba-8481-44f634c81348\" (UID: \"91cb5416-01d5-45ba-8481-44f634c81348\") " Nov 02 22:15:21.319392 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.319281 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91cb5416-01d5-45ba-8481-44f634c81348-maas-api-service-ca" pod "91cb5416-01d5-45ba-8481-44f634c81348" (UID: "91cb5416-01d5-45ba-8481-44f634c81348"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:21.321558 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.321544 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91cb5416-01d5-45ba-8481-44f634c81348-maas-api-service-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:21.334116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.331413 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cb5416-01d5-45ba-8481-44f634c81348-kube-api-access-rb4vj" pod "91cb5416-01d5-45ba-8481-44f634c81348" (UID: "91cb5416-01d5-45ba-8481-44f634c81348"). InnerVolumeSpecName "kube-api-access-rb4vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:21.423935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:21.423902 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb4vj\" (UniqueName: \"kubernetes.io/projected/91cb5416-01d5-45ba-8481-44f634c81348-kube-api-access-rb4vj\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:22.040108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.040077 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" Nov 02 22:15:22.040108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.040097 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7" event={"ID":"91cb5416-01d5-45ba-8481-44f634c81348","Type":"ContainerDied","Data":"04e480d53d317ccdc803d63937bc9026cd3e93c47413375bd11443e142897d80"} Nov 02 22:15:22.040331 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.040204 4309 scope.go:122] "RemoveContainer" containerID="e94b940d732342ef07c0812c1aa048a9955c375f479689d3a919b3ce370e08f1" Nov 02 22:15:22.051488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.051455 4309 generic.go:356] "Generic (PLEG): container finished" podID="90135336-acb5-401c-88ce-457247da8afb" containerID="62887ce05f089fd17fa5eb08c0fde4ca8276cb11787a036e0ba689aa2f298baa" exitCode=60 Nov 02 22:15:22.051625 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.051530 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" event={"ID":"90135336-acb5-401c-88ce-457247da8afb","Type":"ContainerDied","Data":"62887ce05f089fd17fa5eb08c0fde4ca8276cb11787a036e0ba689aa2f298baa"} Nov 02 22:15:22.052076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.052055 4309 scope.go:122] "RemoveContainer" containerID="62887ce05f089fd17fa5eb08c0fde4ca8276cb11787a036e0ba689aa2f298baa" Nov 02 22:15:22.052471 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:22.052443 4309 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6_odh-ai-gateway-infra(90135336-acb5-401c-88ce-457247da8afb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" podUID="90135336-acb5-401c-88ce-457247da8afb" Nov 02 22:15:22.055195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.055177 4309 scope.go:122] "RemoveContainer" containerID="52e877a29dd46f9a347861fceffde802feafffb3a921717d264e4e11e5b14626" Nov 02 22:15:22.089101 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.089069 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6"] Nov 02 22:15:22.159553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.159524 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7"] Nov 02 22:15:22.169426 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.169394 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5f4a2f-29894295-n7mz7"] Nov 02 22:15:22.199279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.199247 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" path="/var/lib/kubelet/pods/6bb8fb52-562c-4c1b-88a8-e1000eba3678/volumes" Nov 02 22:15:22.200212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.200190 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" path="/var/lib/kubelet/pods/8c871e58-223f-425c-8874-63c90a1ec02d/volumes" Nov 02 22:15:22.200984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.200965 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91cb5416-01d5-45ba-8481-44f634c81348" path="/var/lib/kubelet/pods/91cb5416-01d5-45ba-8481-44f634c81348/volumes" Nov 02 22:15:22.257137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.257114 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:22.387775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.363309 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-kube-api-access-8hdqs\" (UniqueName: \"kubernetes.io/projected/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-kube-api-access-8hdqs\") pod \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\" (UID: \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\") " Nov 02 22:15:22.387775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.363481 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-maas-api-service-ca\") pod \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\" (UID: \"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff\") " Nov 02 22:15:22.495698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.457958 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-maas-api-service-ca" pod "8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" (UID: "8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:22.495698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.481939 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-kube-api-access-8hdqs" pod "8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" (UID: "8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff"). InnerVolumeSpecName "kube-api-access-8hdqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:22.504862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.504822 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-maas-api-service-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:22.505092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:22.505073 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hdqs\" (UniqueName: \"kubernetes.io/projected/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff-kube-api-access-8hdqs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:23.033002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.032966 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw"] Nov 02 22:15:23.041943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.041908 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd2315f9-lhqxw"] Nov 02 22:15:23.231756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.224533 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" event={"ID":"8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff","Type":"ContainerDied","Data":"b9e807540b7be75b5427ee2ba7078906e981294665e5ec52b923e3ab16d19f28"} Nov 02 22:15:23.231756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.224605 4309 scope.go:122] "RemoveContainer" containerID="3c481d4f66668cff991642f3da90b6505b0be02f8186bcb625cd8680376c13aa" Nov 02 22:15:23.231756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.224712 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48" Nov 02 22:15:23.358908 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:23.358868 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:23.380764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.380723 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48"] Nov 02 22:15:23.413538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.413499 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-10852f9c-29894295-jcv48"] Nov 02 22:15:23.775326 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.774928 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:23.873411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.873356 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90135336-acb5-401c-88ce-457247da8afb-kube-api-access-5hptz\" (UniqueName: \"kubernetes.io/projected/90135336-acb5-401c-88ce-457247da8afb-kube-api-access-5hptz\") pod \"90135336-acb5-401c-88ce-457247da8afb\" (UID: \"90135336-acb5-401c-88ce-457247da8afb\") " Nov 02 22:15:23.873985 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.873958 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90135336-acb5-401c-88ce-457247da8afb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90135336-acb5-401c-88ce-457247da8afb-maas-api-service-ca\") pod \"90135336-acb5-401c-88ce-457247da8afb\" (UID: \"90135336-acb5-401c-88ce-457247da8afb\") " Nov 02 22:15:23.875222 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.875198 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90135336-acb5-401c-88ce-457247da8afb-maas-api-service-ca" pod "90135336-acb5-401c-88ce-457247da8afb" (UID: "90135336-acb5-401c-88ce-457247da8afb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:15:23.880188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.880032 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90135336-acb5-401c-88ce-457247da8afb-kube-api-access-5hptz" pod "90135336-acb5-401c-88ce-457247da8afb" (UID: "90135336-acb5-401c-88ce-457247da8afb"). InnerVolumeSpecName "kube-api-access-5hptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:15:23.977731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.977553 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hptz\" (UniqueName: \"kubernetes.io/projected/90135336-acb5-401c-88ce-457247da8afb-kube-api-access-5hptz\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:23.977731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:23.977606 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90135336-acb5-401c-88ce-457247da8afb-maas-api-service-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:15:24.046306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.046262 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp"] Nov 02 22:15:24.150945 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.150905 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-197d689b-hnlqp"] Nov 02 22:15:24.169435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.169389 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9"] Nov 02 22:15:24.177138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.177108 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8"] Nov 02 22:15:24.197936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.197891 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" path="/var/lib/kubelet/pods/8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff/volumes" Nov 02 22:15:24.198909 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.198888 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e9142f-cfc2-4753-8f05-677b8c627623" path="/var/lib/kubelet/pods/d5e9142f-cfc2-4753-8f05-677b8c627623/volumes" Nov 02 22:15:24.199609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.199590 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabdc1e2-b48e-4979-93de-4589e5c033fd" path="/var/lib/kubelet/pods/eabdc1e2-b48e-4979-93de-4589e5c033fd/volumes" Nov 02 22:15:24.200886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.200868 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs"] Nov 02 22:15:24.201031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.201018 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51fe0b51-sgjl9"] Nov 02 22:15:24.211291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.211271 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a2b4f4-fmdc8"] Nov 02 22:15:24.222711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.222662 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20cded6-8tfvs"] Nov 02 22:15:24.289786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.289575 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" event={"ID":"90135336-acb5-401c-88ce-457247da8afb","Type":"ContainerDied","Data":"66ffd386e13e1af5a063dac1f4a5c3c34092e1ad95ecc370679dc9980346b484"} Nov 02 22:15:24.289786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.289646 4309 scope.go:122] "RemoveContainer" containerID="62887ce05f089fd17fa5eb08c0fde4ca8276cb11787a036e0ba689aa2f298baa" Nov 02 22:15:24.290058 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.289838 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6" Nov 02 22:15:24.335728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.333849 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6"] Nov 02 22:15:24.344136 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:24.344105 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-310bb3-29894295-stxc6"] Nov 02 22:15:26.320219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.317556 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:15:26.320219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.318224 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:15:26.352797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.352743 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:15:26.435064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.435027 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:15:26.554079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.554038 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ae7159-95e3-478f-b026-3e0335f441a5" path="/var/lib/kubelet/pods/47ae7159-95e3-478f-b026-3e0335f441a5/volumes" Nov 02 22:15:26.569741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.569704 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e73dbf9-dc28-4828-ba5f-08acf8bae7da" path="/var/lib/kubelet/pods/5e73dbf9-dc28-4828-ba5f-08acf8bae7da/volumes" Nov 02 22:15:26.585765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.570725 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906" path="/var/lib/kubelet/pods/73f3aa9b-f01f-49c5-87f8-4d3e5ba1a906/volumes" Nov 02 22:15:26.663711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.652636 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90135336-acb5-401c-88ce-457247da8afb" path="/var/lib/kubelet/pods/90135336-acb5-401c-88ce-457247da8afb/volumes" Nov 02 22:15:26.673056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.664040 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:15:26.689406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.674264 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:15:26.920761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.916763 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:15:26.956327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:26.956283 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:15:33.510636 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:33.510600 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:34.031327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:34.031284 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm"] Nov 02 22:15:34.047126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:34.046757 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h"] Nov 02 22:15:34.051755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:34.051669 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe734ffb-nrnqm"] Nov 02 22:15:34.059474 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:34.059441 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91259f86-kxq5h"] Nov 02 22:15:34.198935 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:34.198893 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135d60c1-4a7f-410d-962e-fe251e968897" path="/var/lib/kubelet/pods/135d60c1-4a7f-410d-962e-fe251e968897/volumes" Nov 02 22:15:34.199759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:34.199722 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6a24fc-3e68-4878-84bc-6051e9d2c1e0" path="/var/lib/kubelet/pods/9e6a24fc-3e68-4878-84bc-6051e9d2c1e0/volumes" Nov 02 22:15:35.890298 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:35.890260 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:36.030665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:36.030617 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2"] Nov 02 22:15:36.036793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:36.036763 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp"] Nov 02 22:15:36.051486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:36.051446 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr"] Nov 02 22:15:36.059392 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:36.059338 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-164d9ca2-qfzx2"] Nov 02 22:15:36.065461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:36.065434 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-587441d6-ktvkp"] Nov 02 22:15:36.070772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:36.070748 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d89cfde-jbhvr"] Nov 02 22:15:36.192954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:36.192876 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df1e741-13d0-4261-a1bb-4a0758d646cc" path="/var/lib/kubelet/pods/0df1e741-13d0-4261-a1bb-4a0758d646cc/volumes" Nov 02 22:15:36.193528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:36.193501 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d" path="/var/lib/kubelet/pods/a7d19c39-b1cf-49e8-ba55-2cf8b2d2f63d/volumes" Nov 02 22:15:36.194055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:36.194038 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01bc386-6c4e-42f3-9383-44b6ae4e6652" path="/var/lib/kubelet/pods/f01bc386-6c4e-42f3-9383-44b6ae4e6652/volumes" Nov 02 22:15:38.027971 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:38.027933 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z"] Nov 02 22:15:38.034636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:38.034605 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b1fd6e-qhf7z"] Nov 02 22:15:38.191527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:38.191496 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267967a3-2066-4b27-9afa-9b8cdab1788a" path="/var/lib/kubelet/pods/267967a3-2066-4b27-9afa-9b8cdab1788a/volumes" Nov 02 22:15:39.024229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:39.024191 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s"] Nov 02 22:15:39.030211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:39.030184 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e7ca20e-8jb2s"] Nov 02 22:15:40.192937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:40.192898 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e8a5ef-1a26-4024-abca-6abe843850bf" path="/var/lib/kubelet/pods/40e8a5ef-1a26-4024-abca-6abe843850bf/volumes" Nov 02 22:15:41.599036 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:41.598997 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:15:43.527145 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:43.527106 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:46.026808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:46.026772 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f"] Nov 02 22:15:46.032898 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:46.032867 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73a2d826-nhl6f"] Nov 02 22:15:46.194552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:46.194518 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62f55ca6-8893-47ce-a6ed-c1357d1159ef" path="/var/lib/kubelet/pods/62f55ca6-8893-47ce-a6ed-c1357d1159ef/volumes" Nov 02 22:15:49.027158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:49.027110 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l"] Nov 02 22:15:49.033621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:49.033597 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c2513-x2m7l"] Nov 02 22:15:50.036956 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.036919 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj"] Nov 02 22:15:50.044303 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.044267 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e806376-8ghhj"] Nov 02 22:15:50.050190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.050162 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b"] Nov 02 22:15:50.057279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.057250 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62"] Nov 02 22:15:50.063394 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.063367 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fea7b4c4-qwk62"] Nov 02 22:15:50.068835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.068810 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5426dfd9-pbw9b"] Nov 02 22:15:50.194386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.194350 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b6d07b-1016-49c7-a472-99f67b47f04b" path="/var/lib/kubelet/pods/24b6d07b-1016-49c7-a472-99f67b47f04b/volumes" Nov 02 22:15:50.195112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.195090 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ceaf50-e4ab-4ccb-b5b5-fad5e395669d" path="/var/lib/kubelet/pods/27ceaf50-e4ab-4ccb-b5b5-fad5e395669d/volumes" Nov 02 22:15:50.195770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.195752 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ea6617-8b17-44c5-825c-2f5e4d80f388" path="/var/lib/kubelet/pods/71ea6617-8b17-44c5-825c-2f5e4d80f388/volumes" Nov 02 22:15:50.196805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:50.196785 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779db660-69dc-4cf4-a004-5b68ba24cdea" path="/var/lib/kubelet/pods/779db660-69dc-4cf4-a004-5b68ba24cdea/volumes" Nov 02 22:15:50.975714 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:15:50.975664 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f299ed_7957_4ade_9951_36e5f02ecc0c.slice\": RecentStats: unable to find data in memory cache]" Nov 02 22:15:59.035748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:59.035712 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz"] Nov 02 22:15:59.041457 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:15:59.041428 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e96973-gs8pz"] Nov 02 22:16:00.195056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:00.195016 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2168bc7f-1970-46a2-9022-45b8704c2975" path="/var/lib/kubelet/pods/2168bc7f-1970-46a2-9022-45b8704c2975/volumes" Nov 02 22:16:02.027350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:02.027322 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl"] Nov 02 22:16:02.035257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:02.035229 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f49131a6-gdqzl"] Nov 02 22:16:02.196236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:02.196201 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c93d6cc-f250-4924-ab3a-d3f3a7855b93" path="/var/lib/kubelet/pods/0c93d6cc-f250-4924-ab3a-d3f3a7855b93/volumes" Nov 02 22:16:03.027559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:03.027525 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz"] Nov 02 22:16:03.033489 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:03.033457 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cb2fc2-qzkhz"] Nov 02 22:16:04.191967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:04.191932 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dfed3e1-ca85-4586-ac5f-021853d85d90" path="/var/lib/kubelet/pods/7dfed3e1-ca85-4586-ac5f-021853d85d90/volumes" Nov 02 22:16:05.939581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.939546 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx"] Nov 02 22:16:05.940317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940302 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerName="cleanup" Nov 02 22:16:05.940367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940320 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerName="cleanup" Nov 02 22:16:05.940367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940331 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" containerName="cleanup" Nov 02 22:16:05.940367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940338 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" containerName="cleanup" Nov 02 22:16:05.940367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940345 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerName="cleanup" Nov 02 22:16:05.940367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940351 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerName="cleanup" Nov 02 22:16:05.940367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940361 4309 cpu_manager.go:414] "containerMap: removing container" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerName="cleanup" Nov 02 22:16:05.940367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940366 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940376 4309 cpu_manager.go:414] "containerMap: removing container" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940390 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940403 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940408 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940415 4309 cpu_manager.go:414] "containerMap: removing container" podUID="90135336-acb5-401c-88ce-457247da8afb" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940421 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90135336-acb5-401c-88ce-457247da8afb" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940428 4309 cpu_manager.go:414] "containerMap: removing container" podUID="91cb5416-01d5-45ba-8481-44f634c81348" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940433 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cb5416-01d5-45ba-8481-44f634c81348" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940438 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940444 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940457 4309 cpu_manager.go:414] "containerMap: removing container" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940461 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940471 4309 cpu_manager.go:414] "containerMap: removing container" podUID="af95b18e-259a-4658-b937-f26fe81fbd20" containerName="revoke-keys" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940475 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af95b18e-259a-4658-b937-f26fe81fbd20" containerName="revoke-keys" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940484 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940489 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940495 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940499 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940512 4309 cpu_manager.go:414] "containerMap: removing container" podUID="91cb5416-01d5-45ba-8481-44f634c81348" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940516 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cb5416-01d5-45ba-8481-44f634c81348" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940524 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940529 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940539 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" containerName="istio-proxy" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940543 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" containerName="istio-proxy" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940549 4309 cpu_manager.go:414] "containerMap: removing container" podUID="90135336-acb5-401c-88ce-457247da8afb" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940554 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90135336-acb5-401c-88ce-457247da8afb" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940567 4309 cpu_manager.go:414] "containerMap: removing container" podUID="90135336-acb5-401c-88ce-457247da8afb" containerName="cleanup" Nov 02 22:16:05.940579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940571 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90135336-acb5-401c-88ce-457247da8afb" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940704 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cb5416-01d5-45ba-8481-44f634c81348" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940714 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940721 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940726 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940731 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940737 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940742 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940746 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="90135336-acb5-401c-88ce-457247da8afb" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940751 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cb5416-01d5-45ba-8481-44f634c81348" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940758 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940763 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb8fb52-562c-4c1b-88a8-e1000eba3678" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940769 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="af95b18e-259a-4658-b937-f26fe81fbd20" containerName="revoke-keys" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940775 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cb5416-01d5-45ba-8481-44f634c81348" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940780 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940786 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00ef6bf-24d1-4cfb-bc8b-9947b36a7a19" containerName="istio-proxy" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940792 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="90135336-acb5-401c-88ce-457247da8afb" containerName="cleanup" Nov 02 22:16:05.941507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.940799 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="90135336-acb5-401c-88ce-457247da8afb" containerName="cleanup" Nov 02 22:16:05.947836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.947810 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:05.952376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.952353 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:16:05.953035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.953015 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:16:05.959255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.954840 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:16:05.959657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:05.959600 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:16:06.007075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.007035 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx"] Nov 02 22:16:06.055739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.055703 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bn8\" (UniqueName: \"kubernetes.io/projected/fa872a23-6787-4da9-a281-93d7057016ef-kube-api-access-46bn8\") pod \"maas-api-revoke-sub-97295b29-c9qzx\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.056063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.056036 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97295b29-c9qzx\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.056250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.056201 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97295b29-c9qzx\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.157525 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.157491 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bn8\" (UniqueName: \"kubernetes.io/projected/fa872a23-6787-4da9-a281-93d7057016ef-kube-api-access-46bn8\") pod \"maas-api-revoke-sub-97295b29-c9qzx\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.157717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.157616 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97295b29-c9qzx\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.157795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.157781 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97295b29-c9qzx\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.158397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.158374 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97295b29-c9qzx\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.160053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.160031 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97295b29-c9qzx\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.165119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.165102 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bn8\" (UniqueName: \"kubernetes.io/projected/fa872a23-6787-4da9-a281-93d7057016ef-kube-api-access-46bn8\") pod \"maas-api-revoke-sub-97295b29-c9qzx\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.333039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.333005 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:06.625075 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:16:06.625038 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa872a23_6787_4da9_a281_93d7057016ef.slice/crio-22ae3610683f5ad8568b75056e33118b6e3784e2d069fd018a77ecbb23350933 WatchSource:0}: Error finding container 22ae3610683f5ad8568b75056e33118b6e3784e2d069fd018a77ecbb23350933: Status 404 returned error can't find the container with id 22ae3610683f5ad8568b75056e33118b6e3784e2d069fd018a77ecbb23350933 Nov 02 22:16:06.625491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:06.625464 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx"] Nov 02 22:16:07.450697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.450650 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" event={"ID":"fa872a23-6787-4da9-a281-93d7057016ef","Type":"ContainerStarted","Data":"3096a4177c6f60653c6c948f4a5a46fd05bc06a99a50a50047d63b75f290345e"} Nov 02 22:16:07.451129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.450724 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" event={"ID":"fa872a23-6787-4da9-a281-93d7057016ef","Type":"ContainerStarted","Data":"22ae3610683f5ad8568b75056e33118b6e3784e2d069fd018a77ecbb23350933"} Nov 02 22:16:07.472425 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.472377 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" podStartSLOduration=2.472362283 podStartE2EDuration="2.472362283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:07.466555111 +0000 UTC m=+1095.404668335" watchObservedRunningTime="2026-11-02 22:16:07.472362283 +0000 UTC m=+1095.410475501" Nov 02 22:16:07.830155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.830115 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9"] Nov 02 22:16:07.831310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.831288 4309 cpu_manager.go:414] "containerMap: removing container" podUID="91cb5416-01d5-45ba-8481-44f634c81348" containerName="cleanup" Nov 02 22:16:07.831387 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.831314 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cb5416-01d5-45ba-8481-44f634c81348" containerName="cleanup" Nov 02 22:16:07.831431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.831391 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" containerName="cleanup" Nov 02 22:16:07.831431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.831399 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" containerName="cleanup" Nov 02 22:16:07.831564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.831552 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c871e58-223f-425c-8874-63c90a1ec02d" containerName="cleanup" Nov 02 22:16:07.831599 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.831574 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f78b57-bc54-4aee-a235-0e4f0da0ffa5" containerName="cleanup" Nov 02 22:16:07.831631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.831596 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1072d5-ec3e-4ba9-b9e4-ee6021d1faff" containerName="cleanup" Nov 02 22:16:07.836343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.836305 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:07.840064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.839918 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9"] Nov 02 22:16:07.889576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.889530 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:07.889767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.889719 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzt8v\" (UniqueName: \"kubernetes.io/projected/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-kube-api-access-tzt8v\") pod \"maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:07.889767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.889762 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:07.991252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.991220 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:07.991424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.991350 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzt8v\" (UniqueName: \"kubernetes.io/projected/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-kube-api-access-tzt8v\") pod \"maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:07.991424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.991382 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:07.992203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.992172 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:07.994232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.994207 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:07.998920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:07.998897 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzt8v\" (UniqueName: \"kubernetes.io/projected/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-kube-api-access-tzt8v\") pod \"maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:08.154716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:08.154592 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:08.549698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:08.534855 4309 generic.go:356] "Generic (PLEG): container finished" podID="fa872a23-6787-4da9-a281-93d7057016ef" containerID="3096a4177c6f60653c6c948f4a5a46fd05bc06a99a50a50047d63b75f290345e" exitCode=0 Nov 02 22:16:08.549698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:08.535275 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" event={"ID":"fa872a23-6787-4da9-a281-93d7057016ef","Type":"ContainerDied","Data":"3096a4177c6f60653c6c948f4a5a46fd05bc06a99a50a50047d63b75f290345e"} Nov 02 22:16:08.549698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:08.548541 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9"] Nov 02 22:16:09.546628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.546585 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" event={"ID":"550f2397-13f2-4a50-8563-f8aa7f4e5e7c","Type":"ContainerStarted","Data":"da731014de462a2387db1a9f00aa54648bdd85e97a6bcfb788d795e67c741db3"} Nov 02 22:16:09.546930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.546651 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" event={"ID":"550f2397-13f2-4a50-8563-f8aa7f4e5e7c","Type":"ContainerStarted","Data":"019b6fc7807ec48b4b7a44bcfcdabdb644b171e6aa670887c2b3c85edb245331"} Nov 02 22:16:09.568926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.568863 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" podStartSLOduration=2.568843942 podStartE2EDuration="2.568843942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:09.560766717 +0000 UTC m=+1097.498879936" watchObservedRunningTime="2026-11-02 22:16:09.568843942 +0000 UTC m=+1097.506957160" Nov 02 22:16:09.753640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.753614 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:09.906312 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.906272 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-xks-ca\") pod \"fa872a23-6787-4da9-a281-93d7057016ef\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " Nov 02 22:16:09.906627 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.906426 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa872a23-6787-4da9-a281-93d7057016ef-kube-api-access-46bn8\" (UniqueName: \"kubernetes.io/projected/fa872a23-6787-4da9-a281-93d7057016ef-kube-api-access-46bn8\") pod \"fa872a23-6787-4da9-a281-93d7057016ef\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " Nov 02 22:16:09.906627 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.906468 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-openshift-ca\") pod \"fa872a23-6787-4da9-a281-93d7057016ef\" (UID: \"fa872a23-6787-4da9-a281-93d7057016ef\") " Nov 02 22:16:09.907635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.907271 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-openshift-ca" pod "fa872a23-6787-4da9-a281-93d7057016ef" (UID: "fa872a23-6787-4da9-a281-93d7057016ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:09.907635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.907601 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:09.909293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.909265 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa872a23-6787-4da9-a281-93d7057016ef-kube-api-access-46bn8" pod "fa872a23-6787-4da9-a281-93d7057016ef" (UID: "fa872a23-6787-4da9-a281-93d7057016ef"). InnerVolumeSpecName "kube-api-access-46bn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:09.909534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:09.909512 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-xks-ca" pod "fa872a23-6787-4da9-a281-93d7057016ef" (UID: "fa872a23-6787-4da9-a281-93d7057016ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:10.009516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:10.009473 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bn8\" (UniqueName: \"kubernetes.io/projected/fa872a23-6787-4da9-a281-93d7057016ef-kube-api-access-46bn8\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:10.009516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:10.009511 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa872a23-6787-4da9-a281-93d7057016ef-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:10.565039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:10.565003 4309 generic.go:356] "Generic (PLEG): container finished" podID="550f2397-13f2-4a50-8563-f8aa7f4e5e7c" containerID="da731014de462a2387db1a9f00aa54648bdd85e97a6bcfb788d795e67c741db3" exitCode=0 Nov 02 22:16:10.565226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:10.565088 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" event={"ID":"550f2397-13f2-4a50-8563-f8aa7f4e5e7c","Type":"ContainerDied","Data":"da731014de462a2387db1a9f00aa54648bdd85e97a6bcfb788d795e67c741db3"} Nov 02 22:16:10.566811 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:10.566787 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" Nov 02 22:16:10.566950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:10.566807 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx" event={"ID":"fa872a23-6787-4da9-a281-93d7057016ef","Type":"ContainerDied","Data":"22ae3610683f5ad8568b75056e33118b6e3784e2d069fd018a77ecbb23350933"} Nov 02 22:16:10.566950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:10.566850 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ae3610683f5ad8568b75056e33118b6e3784e2d069fd018a77ecbb23350933" Nov 02 22:16:11.606046 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:16:11.605986 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:16:11.779717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:11.779671 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:11.953814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:11.953719 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-xks-ca\") pod \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " Nov 02 22:16:11.953968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:11.953903 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-openshift-ca\") pod \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " Nov 02 22:16:11.954018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:11.953972 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-kube-api-access-tzt8v\" (UniqueName: \"kubernetes.io/projected/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-kube-api-access-tzt8v\") pod \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\" (UID: \"550f2397-13f2-4a50-8563-f8aa7f4e5e7c\") " Nov 02 22:16:11.954387 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:11.954346 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-openshift-ca" pod "550f2397-13f2-4a50-8563-f8aa7f4e5e7c" (UID: "550f2397-13f2-4a50-8563-f8aa7f4e5e7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:11.954972 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:11.954945 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:11.956377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:11.956070 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-xks-ca" pod "550f2397-13f2-4a50-8563-f8aa7f4e5e7c" (UID: "550f2397-13f2-4a50-8563-f8aa7f4e5e7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:11.956793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:11.956768 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-kube-api-access-tzt8v" pod "550f2397-13f2-4a50-8563-f8aa7f4e5e7c" (UID: "550f2397-13f2-4a50-8563-f8aa7f4e5e7c"). InnerVolumeSpecName "kube-api-access-tzt8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:12.057134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:12.057099 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzt8v\" (UniqueName: \"kubernetes.io/projected/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-kube-api-access-tzt8v\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:12.057355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:12.057332 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/550f2397-13f2-4a50-8563-f8aa7f4e5e7c-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:12.597313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:12.597271 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" event={"ID":"550f2397-13f2-4a50-8563-f8aa7f4e5e7c","Type":"ContainerDied","Data":"019b6fc7807ec48b4b7a44bcfcdabdb644b171e6aa670887c2b3c85edb245331"} Nov 02 22:16:12.597535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:12.597338 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019b6fc7807ec48b4b7a44bcfcdabdb644b171e6aa670887c2b3c85edb245331" Nov 02 22:16:12.597535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:12.597434 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9" Nov 02 22:16:15.025221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:15.025163 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph"] Nov 02 22:16:15.030917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:15.030888 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a279d67-bbzph"] Nov 02 22:16:16.193387 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:16.193343 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd30d87-804c-4620-8d90-a9448b1d8437" path="/var/lib/kubelet/pods/0dd30d87-804c-4620-8d90-a9448b1d8437/volumes" Nov 02 22:16:25.721805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:25.721763 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9"] Nov 02 22:16:25.793708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:25.780969 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-10852f9c-lhdf9"] Nov 02 22:16:26.201398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:26.201354 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550f2397-13f2-4a50-8563-f8aa7f4e5e7c" path="/var/lib/kubelet/pods/550f2397-13f2-4a50-8563-f8aa7f4e5e7c/volumes" Nov 02 22:16:33.122018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:33.121224 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl"] Nov 02 22:16:33.131537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:33.121938 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" containerID="cri-o://8fd577ea6d5363cc23a0285127405b58b6bf01ac2b31deeb154844b8c0ef89ac" gracePeriod=30 Nov 02 22:16:36.130948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.130561 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww"] Nov 02 22:16:36.218412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.218379 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-149a37c2-jftww"] Nov 02 22:16:36.611789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.611745 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9"] Nov 02 22:16:36.638938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.620944 4309 cpu_manager.go:414] "containerMap: removing container" podUID="fa872a23-6787-4da9-a281-93d7057016ef" containerName="revoke-keys" Nov 02 22:16:36.639169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.639141 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa872a23-6787-4da9-a281-93d7057016ef" containerName="revoke-keys" Nov 02 22:16:36.642698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.639379 4309 cpu_manager.go:414] "containerMap: removing container" podUID="550f2397-13f2-4a50-8563-f8aa7f4e5e7c" containerName="revoke-keys" Nov 02 22:16:36.646785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.646759 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550f2397-13f2-4a50-8563-f8aa7f4e5e7c" containerName="revoke-keys" Nov 02 22:16:36.670165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.670135 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="550f2397-13f2-4a50-8563-f8aa7f4e5e7c" containerName="revoke-keys" Nov 02 22:16:36.670348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.670335 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa872a23-6787-4da9-a281-93d7057016ef" containerName="revoke-keys" Nov 02 22:16:36.768745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.677010 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:36.768745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.678974 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9"] Nov 02 22:16:36.768745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.680666 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25e1cfae-bfvb9\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:36.875438 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.875341 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqmp\" (UniqueName: \"kubernetes.io/projected/48e05464-47f8-4403-9d22-4cd6091c5b90-kube-api-access-fbqmp\") pod \"maas-api-revoke-sub-25e1cfae-bfvb9\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:36.875865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.875824 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25e1cfae-bfvb9\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:36.876190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.876171 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25e1cfae-bfvb9\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:36.913157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.913128 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:16:36.958987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.958955 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:16:36.959314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.959292 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:16:37.115758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:36.964830 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:16:37.204795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.193021 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25e1cfae-bfvb9\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:37.319582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.319535 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:37.512889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.344219 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:37.612361 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.418414 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:37.627701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.427822 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:16:37.627701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.530875 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqmp\" (UniqueName: \"kubernetes.io/projected/48e05464-47f8-4403-9d22-4cd6091c5b90-kube-api-access-fbqmp\") pod \"maas-api-revoke-sub-25e1cfae-bfvb9\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:37.627701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.615715 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25e1cfae-bfvb9\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:37.681974 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.678025 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:37.710367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.708847 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:37.736475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.736441 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqmp\" (UniqueName: \"kubernetes.io/projected/48e05464-47f8-4403-9d22-4cd6091c5b90-kube-api-access-fbqmp\") pod \"maas-api-revoke-sub-25e1cfae-bfvb9\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:37.778749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.775308 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25e1cfae-bfvb9\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:37.886878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.873136 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:37.886878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:37.882765 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:38.003693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:38.003646 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:38.123180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:38.122464 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:16:38.216618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:38.216576 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329404cd-c051-485d-8b30-1e2f949e8ca9" path="/var/lib/kubelet/pods/329404cd-c051-485d-8b30-1e2f949e8ca9/volumes" Nov 02 22:16:39.158718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.158669 4309 generic.go:356] "Generic (PLEG): container finished" podID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerID="8fd577ea6d5363cc23a0285127405b58b6bf01ac2b31deeb154844b8c0ef89ac" exitCode=0 Nov 02 22:16:39.159226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.158833 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" event={"ID":"2972de31-06c1-4022-a3ab-3796c5ecd136","Type":"ContainerDied","Data":"8fd577ea6d5363cc23a0285127405b58b6bf01ac2b31deeb154844b8c0ef89ac"} Nov 02 22:16:39.159456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.159442 4309 scope.go:122] "RemoveContainer" containerID="51189d2b30d4cd50e2c713d1a0f4a1749e3201feed477b5ebeca4e3690e69142" Nov 02 22:16:39.169983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.169954 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9"] Nov 02 22:16:39.314941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.314913 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:16:39.348942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.348906 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-kube-api-access-jvn2p\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-kube-api-access-jvn2p\") pod \"2972de31-06c1-4022-a3ab-3796c5ecd136\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " Nov 02 22:16:39.349139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.348957 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-socket\") pod \"2972de31-06c1-4022-a3ab-3796c5ecd136\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " Nov 02 22:16:39.349139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.349131 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2972de31-06c1-4022-a3ab-3796c5ecd136-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2972de31-06c1-4022-a3ab-3796c5ecd136-istiod-ca-cert\") pod \"2972de31-06c1-4022-a3ab-3796c5ecd136\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " Nov 02 22:16:39.349273 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.349207 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-envoy\") pod \"2972de31-06c1-4022-a3ab-3796c5ecd136\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " Nov 02 22:16:39.349273 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.349238 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-data\") pod \"2972de31-06c1-4022-a3ab-3796c5ecd136\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " Nov 02 22:16:39.349405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.349271 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-podinfo\") pod \"2972de31-06c1-4022-a3ab-3796c5ecd136\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " Nov 02 22:16:39.349405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.349390 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-credential-socket\") pod \"2972de31-06c1-4022-a3ab-3796c5ecd136\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " Nov 02 22:16:39.349512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.349480 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-token\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-token\") pod \"2972de31-06c1-4022-a3ab-3796c5ecd136\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " Nov 02 22:16:39.349576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.349545 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-certs\") pod \"2972de31-06c1-4022-a3ab-3796c5ecd136\" (UID: \"2972de31-06c1-4022-a3ab-3796c5ecd136\") " Nov 02 22:16:39.350165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.349974 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-credential-socket" pod "2972de31-06c1-4022-a3ab-3796c5ecd136" (UID: "2972de31-06c1-4022-a3ab-3796c5ecd136"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:39.350285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.350198 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-certs" pod "2972de31-06c1-4022-a3ab-3796c5ecd136" (UID: "2972de31-06c1-4022-a3ab-3796c5ecd136"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:39.351074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.350762 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:39.351074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.350792 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:39.352232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.351896 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-kube-api-access-jvn2p" pod "2972de31-06c1-4022-a3ab-3796c5ecd136" (UID: "2972de31-06c1-4022-a3ab-3796c5ecd136"). InnerVolumeSpecName "kube-api-access-jvn2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:39.352232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.352192 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-socket" pod "2972de31-06c1-4022-a3ab-3796c5ecd136" (UID: "2972de31-06c1-4022-a3ab-3796c5ecd136"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:39.352571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.352546 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2972de31-06c1-4022-a3ab-3796c5ecd136-istiod-ca-cert" pod "2972de31-06c1-4022-a3ab-3796c5ecd136" (UID: "2972de31-06c1-4022-a3ab-3796c5ecd136"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:39.352873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.352839 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-data" pod "2972de31-06c1-4022-a3ab-3796c5ecd136" (UID: "2972de31-06c1-4022-a3ab-3796c5ecd136"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:39.356128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.355815 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-envoy" pod "2972de31-06c1-4022-a3ab-3796c5ecd136" (UID: "2972de31-06c1-4022-a3ab-3796c5ecd136"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:16:39.356128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.355821 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-podinfo" pod "2972de31-06c1-4022-a3ab-3796c5ecd136" (UID: "2972de31-06c1-4022-a3ab-3796c5ecd136"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:16:39.356128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.355903 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-token" pod "2972de31-06c1-4022-a3ab-3796c5ecd136" (UID: "2972de31-06c1-4022-a3ab-3796c5ecd136"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:39.452402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.452369 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:39.452402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.452399 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:39.452402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.452409 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:39.452636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.452418 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:39.452636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.452427 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvn2p\" (UniqueName: \"kubernetes.io/projected/2972de31-06c1-4022-a3ab-3796c5ecd136-kube-api-access-jvn2p\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:39.452636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.452437 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2972de31-06c1-4022-a3ab-3796c5ecd136-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:39.452636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:39.452445 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2972de31-06c1-4022-a3ab-3796c5ecd136-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:40.187033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:40.187004 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" Nov 02 22:16:40.199251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:40.199217 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" event={"ID":"48e05464-47f8-4403-9d22-4cd6091c5b90","Type":"ContainerStarted","Data":"a5c11ab4844a4c706c6f0e303692fa420c161f3a2b2a80e8828bbed245f44aad"} Nov 02 22:16:40.199251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:40.199265 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" event={"ID":"48e05464-47f8-4403-9d22-4cd6091c5b90","Type":"ContainerStarted","Data":"b19fd198a4028d3f3d6432212fa9d249ff44ebec92fd9d108277d877ca60ea8b"} Nov 02 22:16:40.199473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:40.199287 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl" event={"ID":"2972de31-06c1-4022-a3ab-3796c5ecd136","Type":"ContainerDied","Data":"1bfaf9281091b5ea3b5a57db0b09c53ab9e763b708adbabda91d5966bc61d8a6"} Nov 02 22:16:40.199473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:40.199307 4309 scope.go:122] "RemoveContainer" containerID="8fd577ea6d5363cc23a0285127405b58b6bf01ac2b31deeb154844b8c0ef89ac" Nov 02 22:16:40.203653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:40.203592 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" podStartSLOduration=4.203579172 podStartE2EDuration="4.203579172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:40.203008879 +0000 UTC m=+1128.141122096" watchObservedRunningTime="2026-11-02 22:16:40.203579172 +0000 UTC m=+1128.141692391" Nov 02 22:16:40.246450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:40.246414 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl"] Nov 02 22:16:40.258316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:40.258285 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-10852f9c-openshift-default-574bb94d89-9fwcl"] Nov 02 22:16:41.252660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:41.252075 4309 generic.go:356] "Generic (PLEG): container finished" podID="48e05464-47f8-4403-9d22-4cd6091c5b90" containerID="a5c11ab4844a4c706c6f0e303692fa420c161f3a2b2a80e8828bbed245f44aad" exitCode=0 Nov 02 22:16:41.252660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:41.252151 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" event={"ID":"48e05464-47f8-4403-9d22-4cd6091c5b90","Type":"ContainerDied","Data":"a5c11ab4844a4c706c6f0e303692fa420c161f3a2b2a80e8828bbed245f44aad"} Nov 02 22:16:41.604567 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:16:41.604524 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:16:42.205703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.203237 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" path="/var/lib/kubelet/pods/2972de31-06c1-4022-a3ab-3796c5ecd136/volumes" Nov 02 22:16:42.456909 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.456847 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:42.539997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.539957 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-xks-ca\") pod \"48e05464-47f8-4403-9d22-4cd6091c5b90\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " Nov 02 22:16:42.540191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.540023 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-openshift-ca\") pod \"48e05464-47f8-4403-9d22-4cd6091c5b90\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " Nov 02 22:16:42.540191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.540170 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e05464-47f8-4403-9d22-4cd6091c5b90-kube-api-access-fbqmp\" (UniqueName: \"kubernetes.io/projected/48e05464-47f8-4403-9d22-4cd6091c5b90-kube-api-access-fbqmp\") pod \"48e05464-47f8-4403-9d22-4cd6091c5b90\" (UID: \"48e05464-47f8-4403-9d22-4cd6091c5b90\") " Nov 02 22:16:42.540445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.540409 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-openshift-ca" pod "48e05464-47f8-4403-9d22-4cd6091c5b90" (UID: "48e05464-47f8-4403-9d22-4cd6091c5b90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:42.541296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.541266 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:42.542247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.542221 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e05464-47f8-4403-9d22-4cd6091c5b90-kube-api-access-fbqmp" pod "48e05464-47f8-4403-9d22-4cd6091c5b90" (UID: "48e05464-47f8-4403-9d22-4cd6091c5b90"). InnerVolumeSpecName "kube-api-access-fbqmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:42.542367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.542320 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-xks-ca" pod "48e05464-47f8-4403-9d22-4cd6091c5b90" (UID: "48e05464-47f8-4403-9d22-4cd6091c5b90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:42.643267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.643225 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbqmp\" (UniqueName: \"kubernetes.io/projected/48e05464-47f8-4403-9d22-4cd6091c5b90-kube-api-access-fbqmp\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:42.643267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:42.643269 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e05464-47f8-4403-9d22-4cd6091c5b90-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:43.276034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:43.276000 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" Nov 02 22:16:43.276254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:43.276006 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9" event={"ID":"48e05464-47f8-4403-9d22-4cd6091c5b90","Type":"ContainerDied","Data":"b19fd198a4028d3f3d6432212fa9d249ff44ebec92fd9d108277d877ca60ea8b"} Nov 02 22:16:43.276254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:43.276144 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b19fd198a4028d3f3d6432212fa9d249ff44ebec92fd9d108277d877ca60ea8b" Nov 02 22:16:46.009715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.009660 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l"] Nov 02 22:16:46.046968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.043868 4309 cpu_manager.go:414] "containerMap: removing container" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" Nov 02 22:16:46.046968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.043901 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" Nov 02 22:16:46.046968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.043998 4309 cpu_manager.go:414] "containerMap: removing container" podUID="48e05464-47f8-4403-9d22-4cd6091c5b90" containerName="revoke-keys" Nov 02 22:16:46.046968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.044008 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e05464-47f8-4403-9d22-4cd6091c5b90" containerName="revoke-keys" Nov 02 22:16:46.046968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.044446 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" Nov 02 22:16:46.046968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.044479 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e05464-47f8-4403-9d22-4cd6091c5b90" containerName="revoke-keys" Nov 02 22:16:46.055705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.055105 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l"] Nov 02 22:16:46.055705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.055221 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:46.154701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.154184 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2fc3ce\"" Nov 02 22:16:46.228700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.227968 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2rh\" (UniqueName: \"kubernetes.io/projected/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-kube-api-access-rl2rh\") pod \"payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l\" (UID: \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:46.228700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.228190 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l\" (UID: \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:46.333698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.329794 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2rh\" (UniqueName: \"kubernetes.io/projected/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-kube-api-access-rl2rh\") pod \"payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l\" (UID: \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:46.333698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.330085 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l\" (UID: \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:46.333698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.331266 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l\" (UID: \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:46.370342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.370305 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2rh\" (UniqueName: \"kubernetes.io/projected/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-kube-api-access-rl2rh\") pod \"payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l\" (UID: \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:46.379466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.379431 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:46.705800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:46.705770 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l"] Nov 02 22:16:46.706611 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:16:46.706574 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2411b0_3bee_45dc_8ac7_c9acffe89af0.slice/crio-e5513f3f53b7b7c0cb1e172e655b2bcec09a6229b726edb20e20746a1e65288d WatchSource:0}: Error finding container e5513f3f53b7b7c0cb1e172e655b2bcec09a6229b726edb20e20746a1e65288d: Status 404 returned error can't find the container with id e5513f3f53b7b7c0cb1e172e655b2bcec09a6229b726edb20e20746a1e65288d Nov 02 22:16:47.384504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:47.384467 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" event={"ID":"3f2411b0-3bee-45dc-8ac7-c9acffe89af0","Type":"ContainerStarted","Data":"3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408"} Nov 02 22:16:47.385050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:47.384524 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" event={"ID":"3f2411b0-3bee-45dc-8ac7-c9acffe89af0","Type":"ContainerStarted","Data":"e5513f3f53b7b7c0cb1e172e655b2bcec09a6229b726edb20e20746a1e65288d"} Nov 02 22:16:47.385050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:47.384587 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:47.408257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:47.408203 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" podStartSLOduration=2.4081851260000002 podStartE2EDuration="2.408185126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:47.407454415 +0000 UTC m=+1135.345567634" watchObservedRunningTime="2026-11-02 22:16:47.408185126 +0000 UTC m=+1135.346298345" Nov 02 22:16:52.303845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.303813 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg"] Nov 02 22:16:52.305012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.304986 4309 cpu_manager.go:414] "containerMap: removing container" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" Nov 02 22:16:52.305012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.305008 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" Nov 02 22:16:52.305218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.305157 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="2972de31-06c1-4022-a3ab-3796c5ecd136" containerName="istio-proxy" Nov 02 22:16:52.309382 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.309355 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.313241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.313213 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:16:52.314664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.314640 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:16:52.317994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.317975 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg"] Nov 02 22:16:52.333095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.333066 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:16:52.333235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.333181 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:16:52.367951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.367919 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xtzx\" (UniqueName: \"kubernetes.io/projected/c914f992-2af7-4a09-8c7c-940c4ceac741-kube-api-access-5xtzx\") pod \"maas-api-revoke-sub-2e41425e-zgbhg\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.368244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.368217 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e41425e-zgbhg\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.368452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.368426 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e41425e-zgbhg\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.469930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.469891 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e41425e-zgbhg\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.470183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.470147 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xtzx\" (UniqueName: \"kubernetes.io/projected/c914f992-2af7-4a09-8c7c-940c4ceac741-kube-api-access-5xtzx\") pod \"maas-api-revoke-sub-2e41425e-zgbhg\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.470307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.470251 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e41425e-zgbhg\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.470812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.470790 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e41425e-zgbhg\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.472284 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.472262 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e41425e-zgbhg\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.477818 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.477792 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xtzx\" (UniqueName: \"kubernetes.io/projected/c914f992-2af7-4a09-8c7c-940c4ceac741-kube-api-access-5xtzx\") pod \"maas-api-revoke-sub-2e41425e-zgbhg\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.632620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.632585 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:52.930808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:52.930756 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg"] Nov 02 22:16:52.930936 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:16:52.930835 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc914f992_2af7_4a09_8c7c_940c4ceac741.slice/crio-8b9cae239bf0336706aa498d54e2452e300b3ea3b130e4c23aeffb784bcfcbd5 WatchSource:0}: Error finding container 8b9cae239bf0336706aa498d54e2452e300b3ea3b130e4c23aeffb784bcfcbd5: Status 404 returned error can't find the container with id 8b9cae239bf0336706aa498d54e2452e300b3ea3b130e4c23aeffb784bcfcbd5 Nov 02 22:16:53.476079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.476046 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" event={"ID":"c914f992-2af7-4a09-8c7c-940c4ceac741","Type":"ContainerStarted","Data":"93c6afb0c945ec72242f453b278827d7b4fd0467518b2079a005cc44fa993472"} Nov 02 22:16:53.476469 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.476101 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" event={"ID":"c914f992-2af7-4a09-8c7c-940c4ceac741","Type":"ContainerStarted","Data":"8b9cae239bf0336706aa498d54e2452e300b3ea3b130e4c23aeffb784bcfcbd5"} Nov 02 22:16:53.499502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.499452 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" podStartSLOduration=1.499438356 podStartE2EDuration="1.499438356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:53.492805041 +0000 UTC m=+1141.430918259" watchObservedRunningTime="2026-11-02 22:16:53.499438356 +0000 UTC m=+1141.437551573" Nov 02 22:16:53.552024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.551988 4309 scope.go:122] "RemoveContainer" containerID="02b069cf87e256c2ef56b34c070dba9dc1482f5f98b37334f6b0a923c8a58374" Nov 02 22:16:53.599764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.594971 4309 scope.go:122] "RemoveContainer" containerID="57f33735e211e89cdeee6eb116bb415bb2b6bac4a64532c3bbb727c10085d13c" Nov 02 22:16:53.626972 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.622258 4309 scope.go:122] "RemoveContainer" containerID="c72152dfce2568d0a510f87cff750399b760499670c9c022f71b2d319d4cbc99" Nov 02 22:16:53.650022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.649991 4309 scope.go:122] "RemoveContainer" containerID="bc0a6dbce3fab541c35fce2c88ea8f27746d1c5a7bb69b1a57f0b8b186ecbddf" Nov 02 22:16:53.673887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.670841 4309 scope.go:122] "RemoveContainer" containerID="ea3d153435f738db905e7870668495aa5170e986e21326316b57380cc2dd3c44" Nov 02 22:16:53.685721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.685694 4309 scope.go:122] "RemoveContainer" containerID="ea1e5839def8ad37c0b6045dbdbff175949396a21e3d6b069c2741c9d0763be7" Nov 02 22:16:53.696983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.696962 4309 scope.go:122] "RemoveContainer" containerID="9a1cf2fa345bb4510cd2be2cab687ddeccb24da5b68ccebf1331acc7d7fb1623" Nov 02 22:16:53.709145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.709119 4309 scope.go:122] "RemoveContainer" containerID="34064baa5e5057b5a68b35d1a1461ac0824332cc566fe5d557c585623076b08e" Nov 02 22:16:53.719917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.719890 4309 scope.go:122] "RemoveContainer" containerID="6850c99fd259dbb63ae460d7bf8c443f5689e2185db8d20d98bf8efc6a06dbb4" Nov 02 22:16:53.730715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.730649 4309 scope.go:122] "RemoveContainer" containerID="0934ab8ad24bac6983770609626f59f619eee6dbb72faaf5c77ea2fc1d7240c5" Nov 02 22:16:53.741244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.741226 4309 scope.go:122] "RemoveContainer" containerID="7329f6ee84d03a813bf2f78c4f2f9baffe6a810e2e94cbef95ffaa192f5dc59b" Nov 02 22:16:53.751712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.751651 4309 scope.go:122] "RemoveContainer" containerID="5febeb55bc2929182337f4ea0ef4951dfd14c3ef70c8bfb609f80377f11bfbf9" Nov 02 22:16:53.762177 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.762159 4309 scope.go:122] "RemoveContainer" containerID="6463e53f82e58cc3078bc53c79519e1b3d0787bd92fbb5b564f99997449218b9" Nov 02 22:16:53.772538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.772512 4309 scope.go:122] "RemoveContainer" containerID="e786af591d6e337b84095e660f3d791b761b3d457ac4e56d3e78600e5e8bcae8" Nov 02 22:16:53.784188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.784166 4309 scope.go:122] "RemoveContainer" containerID="11719b3f9063767013525822659eb7f8168afacb19a34340df6d3c3b09bce369" Nov 02 22:16:53.796722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.796700 4309 scope.go:122] "RemoveContainer" containerID="b805d49709dbbffd5acdbad93af9e027aaf48760c8d86762b49f526034fb9363" Nov 02 22:16:53.809902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:53.809884 4309 scope.go:122] "RemoveContainer" containerID="c302d49329a36f2a390b53106ab1c5ef957192ead0577476df7e72e3e95102c2" Nov 02 22:16:54.488721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:54.488668 4309 generic.go:356] "Generic (PLEG): container finished" podID="c914f992-2af7-4a09-8c7c-940c4ceac741" containerID="93c6afb0c945ec72242f453b278827d7b4fd0467518b2079a005cc44fa993472" exitCode=0 Nov 02 22:16:54.488721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:54.488710 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" event={"ID":"c914f992-2af7-4a09-8c7c-940c4ceac741","Type":"ContainerDied","Data":"93c6afb0c945ec72242f453b278827d7b4fd0467518b2079a005cc44fa993472"} Nov 02 22:16:55.730761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.730661 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:55.778858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.778775 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-xks-ca\") pod \"c914f992-2af7-4a09-8c7c-940c4ceac741\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " Nov 02 22:16:55.779017 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.778937 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c914f992-2af7-4a09-8c7c-940c4ceac741-kube-api-access-5xtzx\" (UniqueName: \"kubernetes.io/projected/c914f992-2af7-4a09-8c7c-940c4ceac741-kube-api-access-5xtzx\") pod \"c914f992-2af7-4a09-8c7c-940c4ceac741\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " Nov 02 22:16:55.779254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.779233 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-openshift-ca\") pod \"c914f992-2af7-4a09-8c7c-940c4ceac741\" (UID: \"c914f992-2af7-4a09-8c7c-940c4ceac741\") " Nov 02 22:16:55.779749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.779717 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-openshift-ca" pod "c914f992-2af7-4a09-8c7c-940c4ceac741" (UID: "c914f992-2af7-4a09-8c7c-940c4ceac741"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:16:55.780077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.780054 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:55.782143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.782116 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-xks-ca" pod "c914f992-2af7-4a09-8c7c-940c4ceac741" (UID: "c914f992-2af7-4a09-8c7c-940c4ceac741"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:16:55.782492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.782472 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c914f992-2af7-4a09-8c7c-940c4ceac741-kube-api-access-5xtzx" pod "c914f992-2af7-4a09-8c7c-940c4ceac741" (UID: "c914f992-2af7-4a09-8c7c-940c4ceac741"). InnerVolumeSpecName "kube-api-access-5xtzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:16:55.882736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.882585 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c914f992-2af7-4a09-8c7c-940c4ceac741-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:55.882736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:55.882646 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xtzx\" (UniqueName: \"kubernetes.io/projected/c914f992-2af7-4a09-8c7c-940c4ceac741-kube-api-access-5xtzx\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:16:56.155756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.155722 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f"] Nov 02 22:16:56.156730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.156687 4309 cpu_manager.go:414] "containerMap: removing container" podUID="c914f992-2af7-4a09-8c7c-940c4ceac741" containerName="revoke-keys" Nov 02 22:16:56.156730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.156710 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c914f992-2af7-4a09-8c7c-940c4ceac741" containerName="revoke-keys" Nov 02 22:16:56.156960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.156936 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="c914f992-2af7-4a09-8c7c-940c4ceac741" containerName="revoke-keys" Nov 02 22:16:56.160989 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.160970 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.173433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.173406 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-a3713a-openshift-default-dockercfg-7x8zr\"" Nov 02 22:16:56.199574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.199541 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f"] Nov 02 22:16:56.299643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.299589 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-data\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.301450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.301255 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-certs\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.301450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.301307 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istiod-ca-cert\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.301450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.301365 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-credential-socket\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.302070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.301984 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-socket\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.302404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.302360 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-envoy\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.302638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.302623 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-token\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.305819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.305784 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rrb\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-kube-api-access-87rrb\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.306073 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.306056 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-podinfo\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.414784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.414700 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-data\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.415047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.415024 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-certs\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.415152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.415076 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istiod-ca-cert\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.415152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.415115 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-credential-socket\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.415266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.415225 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-socket\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.415377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.415360 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-envoy\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.415468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.415448 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-token\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.415541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.415516 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rrb\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-kube-api-access-87rrb\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.415667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.415581 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-podinfo\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.416490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.416462 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-credential-socket\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.416764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.416740 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-data\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.417010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.416985 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-certs\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.417816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.417645 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istiod-ca-cert\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.425742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.418987 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-socket\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.429379 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.429327 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-envoy\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.434601 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.434577 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-podinfo\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.443735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.443646 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-token\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.444205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.444179 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rrb\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-kube-api-access-87rrb\") pod \"e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.491878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.491847 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:56.526943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.526916 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" Nov 02 22:16:56.527328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.527297 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg" event={"ID":"c914f992-2af7-4a09-8c7c-940c4ceac741","Type":"ContainerDied","Data":"8b9cae239bf0336706aa498d54e2452e300b3ea3b130e4c23aeffb784bcfcbd5"} Nov 02 22:16:56.527503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.527488 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b9cae239bf0336706aa498d54e2452e300b3ea3b130e4c23aeffb784bcfcbd5" Nov 02 22:16:56.825568 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:16:56.825525 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f9e420_c9ca_4a32_bf79_6d6d62097e9a.slice/crio-5ae8f8da5f3276ff4f0fdcf82963963f7e56bf6c598a1f9f712dab113a27e51c WatchSource:0}: Error finding container 5ae8f8da5f3276ff4f0fdcf82963963f7e56bf6c598a1f9f712dab113a27e51c: Status 404 returned error can't find the container with id 5ae8f8da5f3276ff4f0fdcf82963963f7e56bf6c598a1f9f712dab113a27e51c Nov 02 22:16:56.828123 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.828099 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f"] Nov 02 22:16:56.830402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.830363 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:16:56.830493 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.830453 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:16:56.830533 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:56.830503 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:16:57.541667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:57.541628 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" event={"ID":"04f9e420-c9ca-4a32-bf79-6d6d62097e9a","Type":"ContainerStarted","Data":"facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5"} Nov 02 22:16:57.541667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:57.541738 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" event={"ID":"04f9e420-c9ca-4a32-bf79-6d6d62097e9a","Type":"ContainerStarted","Data":"5ae8f8da5f3276ff4f0fdcf82963963f7e56bf6c598a1f9f712dab113a27e51c"} Nov 02 22:16:57.575348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:57.575277 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" podStartSLOduration=1.5752564279999999 podStartE2EDuration="1.575256428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:16:57.568996746 +0000 UTC m=+1145.507109975" watchObservedRunningTime="2026-11-02 22:16:57.575256428 +0000 UTC m=+1145.513369647" Nov 02 22:16:58.400932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:58.400900 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:16:58.492411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:58.492377 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:58.498228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:58.498199 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:58.552294 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:58.552267 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:16:58.553295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:16:58.553275 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:17:01.139853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.139815 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc"] Nov 02 22:17:01.145432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.145368 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:01.149496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.149443 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-a3713a\"" Nov 02 22:17:01.156565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.156379 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5"] Nov 02 22:17:01.178753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.178722 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc"] Nov 02 22:17:01.178941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.178781 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5"] Nov 02 22:17:01.178941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.178925 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:01.185898 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.183965 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-a3713a-dockercfg-z9s92\"" Nov 02 22:17:01.267691 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.267635 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmkd\" (UniqueName: \"kubernetes.io/projected/42506da4-8df1-4e2c-822b-3463e867de25-kube-api-access-8gmkd\") pod \"payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc\" (UID: \"42506da4-8df1-4e2c-822b-3463e867de25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:01.268286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.267789 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e586569-339e-4d3a-a417-496d2635c13f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5\" (UID: \"4e586569-339e-4d3a-a417-496d2635c13f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:01.268286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.268039 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkvv\" (UniqueName: \"kubernetes.io/projected/4e586569-339e-4d3a-a417-496d2635c13f-kube-api-access-mwkvv\") pod \"payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5\" (UID: \"4e586569-339e-4d3a-a417-496d2635c13f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:01.268286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.268173 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42506da4-8df1-4e2c-822b-3463e867de25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc\" (UID: \"42506da4-8df1-4e2c-822b-3463e867de25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:01.384769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.378827 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42506da4-8df1-4e2c-822b-3463e867de25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc\" (UID: \"42506da4-8df1-4e2c-822b-3463e867de25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:01.384769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.379013 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmkd\" (UniqueName: \"kubernetes.io/projected/42506da4-8df1-4e2c-822b-3463e867de25-kube-api-access-8gmkd\") pod \"payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc\" (UID: \"42506da4-8df1-4e2c-822b-3463e867de25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:01.384769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.379118 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e586569-339e-4d3a-a417-496d2635c13f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5\" (UID: \"4e586569-339e-4d3a-a417-496d2635c13f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:01.384769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.379275 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkvv\" (UniqueName: \"kubernetes.io/projected/4e586569-339e-4d3a-a417-496d2635c13f-kube-api-access-mwkvv\") pod \"payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5\" (UID: \"4e586569-339e-4d3a-a417-496d2635c13f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:01.384769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.380587 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42506da4-8df1-4e2c-822b-3463e867de25-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc\" (UID: \"42506da4-8df1-4e2c-822b-3463e867de25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:01.407752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.396955 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e586569-339e-4d3a-a417-496d2635c13f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5\" (UID: \"4e586569-339e-4d3a-a417-496d2635c13f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:01.441697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.430746 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkvv\" (UniqueName: \"kubernetes.io/projected/4e586569-339e-4d3a-a417-496d2635c13f-kube-api-access-mwkvv\") pod \"payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5\" (UID: \"4e586569-339e-4d3a-a417-496d2635c13f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:01.441697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.433395 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmkd\" (UniqueName: \"kubernetes.io/projected/42506da4-8df1-4e2c-822b-3463e867de25-kube-api-access-8gmkd\") pod \"payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc\" (UID: \"42506da4-8df1-4e2c-822b-3463e867de25\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:01.499741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.494096 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:01.505039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.505007 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:01.848191 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:17:01.848146 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e586569_339e_4d3a_a417_496d2635c13f.slice/crio-9366e852ccc65606479762657c39030c63d1558d2f6887f276e3724e419a8f8a WatchSource:0}: Error finding container 9366e852ccc65606479762657c39030c63d1558d2f6887f276e3724e419a8f8a: Status 404 returned error can't find the container with id 9366e852ccc65606479762657c39030c63d1558d2f6887f276e3724e419a8f8a Nov 02 22:17:01.851712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.851070 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5"] Nov 02 22:17:01.980912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:01.980723 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc"] Nov 02 22:17:02.631580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:02.631542 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" event={"ID":"42506da4-8df1-4e2c-822b-3463e867de25","Type":"ContainerStarted","Data":"ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316"} Nov 02 22:17:02.632064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:02.631603 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" event={"ID":"42506da4-8df1-4e2c-822b-3463e867de25","Type":"ContainerStarted","Data":"504b81f6028a889e53f698d5232ef149730078583c69f7bfddd498ee24ecb10b"} Nov 02 22:17:02.632064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:02.631664 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:02.633420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:02.633397 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" event={"ID":"4e586569-339e-4d3a-a417-496d2635c13f","Type":"ContainerStarted","Data":"fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2"} Nov 02 22:17:02.633561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:02.633446 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" event={"ID":"4e586569-339e-4d3a-a417-496d2635c13f","Type":"ContainerStarted","Data":"9366e852ccc65606479762657c39030c63d1558d2f6887f276e3724e419a8f8a"} Nov 02 22:17:02.633561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:02.633474 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:02.676092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:02.676033 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" podStartSLOduration=1.676013215 podStartE2EDuration="1.676013215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:02.661400237 +0000 UTC m=+1150.599513484" watchObservedRunningTime="2026-11-02 22:17:02.676013215 +0000 UTC m=+1150.614126433" Nov 02 22:17:02.690585 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:02.690481 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" podStartSLOduration=1.690460926 podStartE2EDuration="1.690460926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:02.68532179 +0000 UTC m=+1150.623435008" watchObservedRunningTime="2026-11-02 22:17:02.690460926 +0000 UTC m=+1150.628574145" Nov 02 22:17:05.675724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.675524 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4"] Nov 02 22:17:05.682244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.682211 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.707217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.706692 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0dd701-openshift-default-dockercfg-jqx6p\"" Nov 02 22:17:05.713795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.713766 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4"] Nov 02 22:17:05.800158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.800118 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-data\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.800356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.800194 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-token\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.800356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.800246 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-certs\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.800478 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.800364 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0762470-8657-4a0e-b29f-e6064a65d2ca-istiod-ca-cert\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.800580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.800561 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-podinfo\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.800734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.800714 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b6fc\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-kube-api-access-4b6fc\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.800889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.800749 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-envoy\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.800889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.800767 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-socket\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.800889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.800822 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-credential-socket\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.960749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.938373 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-podinfo\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.961196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.961158 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6fc\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-kube-api-access-4b6fc\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.963376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.963343 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-envoy\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.963580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.963557 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-socket\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.963767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.963749 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-credential-socket\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.964065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.964045 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-data\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.964343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.964322 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-token\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.981941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.981910 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-certs\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:05.994830 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.994797 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0762470-8657-4a0e-b29f-e6064a65d2ca-istiod-ca-cert\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.002088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.966375 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-data\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.003834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.982400 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-certs\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.004006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.965156 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-socket\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.004079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.987020 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-envoy\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.004153 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.987548 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-podinfo\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.004223 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:05.966161 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-credential-socket\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.004292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:06.000885 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0762470-8657-4a0e-b29f-e6064a65d2ca-istiod-ca-cert\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.019078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:06.019047 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6fc\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-kube-api-access-4b6fc\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.039851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:06.039787 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-token\") pod \"e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.316709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:06.316507 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:06.625826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:06.625795 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4"] Nov 02 22:17:06.626830 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:17:06.626786 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0762470_8657_4a0e_b29f_e6064a65d2ca.slice/crio-3c72521e8033bf4d9d2f6b955a132c5b29b31333cbbf63d5e304846d25f9eb28 WatchSource:0}: Error finding container 3c72521e8033bf4d9d2f6b955a132c5b29b31333cbbf63d5e304846d25f9eb28: Status 404 returned error can't find the container with id 3c72521e8033bf4d9d2f6b955a132c5b29b31333cbbf63d5e304846d25f9eb28 Nov 02 22:17:06.631065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:06.631031 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:06.631169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:06.631119 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:06.631243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:06.631169 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:06.685388 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:06.685352 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" event={"ID":"c0762470-8657-4a0e-b29f-e6064a65d2ca","Type":"ContainerStarted","Data":"3c72521e8033bf4d9d2f6b955a132c5b29b31333cbbf63d5e304846d25f9eb28"} Nov 02 22:17:07.629473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.629440 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh"] Nov 02 22:17:07.634761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.634734 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.638407 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.638379 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:17:07.639508 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.639487 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:17:07.640729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.640708 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh"] Nov 02 22:17:07.702736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.702279 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" event={"ID":"c0762470-8657-4a0e-b29f-e6064a65d2ca","Type":"ContainerStarted","Data":"213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29"} Nov 02 22:17:07.705485 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.705454 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:17:07.705485 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.705454 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:17:07.727391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.727331 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podStartSLOduration=2.7273147079999998 podStartE2EDuration="2.727314708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:07.721026752 +0000 UTC m=+1155.659139971" watchObservedRunningTime="2026-11-02 22:17:07.727314708 +0000 UTC m=+1155.665427927" Nov 02 22:17:07.747023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.746995 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-fbpnh\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.747165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.747144 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mms64\" (UniqueName: \"kubernetes.io/projected/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-kube-api-access-mms64\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-fbpnh\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.747350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.747336 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-fbpnh\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.862699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.862096 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-fbpnh\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.862699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.862303 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-fbpnh\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.862699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.862470 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mms64\" (UniqueName: \"kubernetes.io/projected/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-kube-api-access-mms64\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-fbpnh\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.876723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.870410 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-fbpnh\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.890259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.890176 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-fbpnh\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.891917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.891892 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mms64\" (UniqueName: \"kubernetes.io/projected/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-kube-api-access-mms64\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-fbpnh\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:07.953879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:07.953434 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:08.043937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.043896 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj"] Nov 02 22:17:08.054123 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.054092 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f328a37-7nblj"] Nov 02 22:17:08.195655 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.195575 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4264fd4-a060-4e41-b206-c0ae0eda7eef" path="/var/lib/kubelet/pods/b4264fd4-a060-4e41-b206-c0ae0eda7eef/volumes" Nov 02 22:17:08.277418 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:17:08.277389 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6eb61f7_d2c8_4eb4_8dd5_c14f37e5a5dd.slice/crio-c6962dfbbd9190ef365eedd5d584f853a51d3111e7d95d7552998fee62348672 WatchSource:0}: Error finding container c6962dfbbd9190ef365eedd5d584f853a51d3111e7d95d7552998fee62348672: Status 404 returned error can't find the container with id c6962dfbbd9190ef365eedd5d584f853a51d3111e7d95d7552998fee62348672 Nov 02 22:17:08.280061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.280039 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh"] Nov 02 22:17:08.320630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.320514 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:08.341749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.340994 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:08.715321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.715283 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" event={"ID":"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd","Type":"ContainerStarted","Data":"ef3b55788e63bd91f11ea1bd040940a38e2d6fd23f35493f6b9b82a9324ae4d6"} Nov 02 22:17:08.715835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.715345 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" event={"ID":"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd","Type":"ContainerStarted","Data":"c6962dfbbd9190ef365eedd5d584f853a51d3111e7d95d7552998fee62348672"} Nov 02 22:17:08.715835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.715766 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:08.716863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.716842 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:17:08.736352 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:08.736299 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" podStartSLOduration=1.736285106 podStartE2EDuration="1.736285106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:08.730822643 +0000 UTC m=+1156.668935860" watchObservedRunningTime="2026-11-02 22:17:08.736285106 +0000 UTC m=+1156.674398324" Nov 02 22:17:09.731331 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:09.731294 4309 generic.go:356] "Generic (PLEG): container finished" podID="b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd" containerID="ef3b55788e63bd91f11ea1bd040940a38e2d6fd23f35493f6b9b82a9324ae4d6" exitCode=0 Nov 02 22:17:09.732016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:09.731371 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" event={"ID":"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd","Type":"ContainerDied","Data":"ef3b55788e63bd91f11ea1bd040940a38e2d6fd23f35493f6b9b82a9324ae4d6"} Nov 02 22:17:10.998554 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:10.998485 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:11.047697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.044415 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt"] Nov 02 22:17:11.057751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.054469 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m"] Nov 02 22:17:11.072154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.072092 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g"] Nov 02 22:17:11.091542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.082496 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-xks-ca\") pod \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " Nov 02 22:17:11.091542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.082608 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-kube-api-access-mms64\" (UniqueName: \"kubernetes.io/projected/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-kube-api-access-mms64\") pod \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " Nov 02 22:17:11.091542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.082794 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-openshift-ca\") pod \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\" (UID: \"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd\") " Nov 02 22:17:11.091542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.083513 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-openshift-ca" pod "b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd" (UID: "b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:11.091542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.083950 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:11.091542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.085086 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb4393f5-84hkt"] Nov 02 22:17:11.091542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.086397 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-xks-ca" pod "b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd" (UID: "b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:17:11.094186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.094156 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-kube-api-access-mms64" pod "b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd" (UID: "b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd"). InnerVolumeSpecName "kube-api-access-mms64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:11.098731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.098663 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6474c875-ktr2m"] Nov 02 22:17:11.120721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.119062 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c861c0b-vp87g"] Nov 02 22:17:11.185860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.185819 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:11.185860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.185856 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mms64\" (UniqueName: \"kubernetes.io/projected/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd-kube-api-access-mms64\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:11.213869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.213839 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7"] Nov 02 22:17:11.214669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.214653 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd" containerName="revoke-keys" Nov 02 22:17:11.214755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.214670 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd" containerName="revoke-keys" Nov 02 22:17:11.214879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.214869 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd" containerName="revoke-keys" Nov 02 22:17:11.236921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.236886 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:11.241766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.241727 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7"] Nov 02 22:17:11.242169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.242150 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0dd701\"" Nov 02 22:17:11.242635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.242583 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0dd701-dockercfg-4hmqr\"" Nov 02 22:17:11.367703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.317132 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6pfg\" (UniqueName: \"kubernetes.io/projected/ce66a98c-5f8d-4003-a769-9cdd31788f6e-kube-api-access-b6pfg\") pod \"payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7\" (UID: \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:11.394094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.394054 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66a98c-5f8d-4003-a769-9cdd31788f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7\" (UID: \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:11.533781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.530557 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6pfg\" (UniqueName: \"kubernetes.io/projected/ce66a98c-5f8d-4003-a769-9cdd31788f6e-kube-api-access-b6pfg\") pod \"payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7\" (UID: \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:11.533781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.531332 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66a98c-5f8d-4003-a769-9cdd31788f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7\" (UID: \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:11.537694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.534480 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66a98c-5f8d-4003-a769-9cdd31788f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7\" (UID: \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:11.543051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.542765 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6pfg\" (UniqueName: \"kubernetes.io/projected/ce66a98c-5f8d-4003-a769-9cdd31788f6e-kube-api-access-b6pfg\") pod \"payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7\" (UID: \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:11.563706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.562794 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:11.639692 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:17:11.637346 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:17:11.805933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.805843 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" Nov 02 22:17:11.806179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.806137 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh" event={"ID":"b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd","Type":"ContainerDied","Data":"c6962dfbbd9190ef365eedd5d584f853a51d3111e7d95d7552998fee62348672"} Nov 02 22:17:11.806253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.806202 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6962dfbbd9190ef365eedd5d584f853a51d3111e7d95d7552998fee62348672" Nov 02 22:17:11.891335 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:17:11.891306 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce66a98c_5f8d_4003_a769_9cdd31788f6e.slice/crio-c65590cd50e818df700d14b26b9fd116a0f7974e320d6bca4b3be77e145b4685 WatchSource:0}: Error finding container c65590cd50e818df700d14b26b9fd116a0f7974e320d6bca4b3be77e145b4685: Status 404 returned error can't find the container with id c65590cd50e818df700d14b26b9fd116a0f7974e320d6bca4b3be77e145b4685 Nov 02 22:17:11.893355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:11.893332 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7"] Nov 02 22:17:12.193659 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:12.193621 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c28dd1d-d2df-47a7-a1b5-8c15117cde76" path="/var/lib/kubelet/pods/2c28dd1d-d2df-47a7-a1b5-8c15117cde76/volumes" Nov 02 22:17:12.194430 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:12.194410 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2648d5-b7a0-4fa3-a83a-1490ae575afc" path="/var/lib/kubelet/pods/ac2648d5-b7a0-4fa3-a83a-1490ae575afc/volumes" Nov 02 22:17:12.194930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:12.194905 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45f9097-9224-44dc-b3e8-3424cc70e7f8" path="/var/lib/kubelet/pods/f45f9097-9224-44dc-b3e8-3424cc70e7f8/volumes" Nov 02 22:17:12.822269 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:12.822234 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" event={"ID":"ce66a98c-5f8d-4003-a769-9cdd31788f6e","Type":"ContainerStarted","Data":"60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4"} Nov 02 22:17:12.822450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:12.822304 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:12.822450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:12.822327 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" event={"ID":"ce66a98c-5f8d-4003-a769-9cdd31788f6e","Type":"ContainerStarted","Data":"c65590cd50e818df700d14b26b9fd116a0f7974e320d6bca4b3be77e145b4685"} Nov 02 22:17:12.843944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:12.843892 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" podStartSLOduration=1.843877071 podStartE2EDuration="1.843877071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:12.838724073 +0000 UTC m=+1160.776837292" watchObservedRunningTime="2026-11-02 22:17:12.843877071 +0000 UTC m=+1160.781990288" Nov 02 22:17:13.652099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:13.652066 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:17:13.652506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:13.652151 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:17:18.349593 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:18.348445 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:18.503919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:18.481409 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:17:18.624663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:18.598111 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 02 22:17:18.624663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:18.598372 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r5hng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:17:18.624663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:18.598403 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:18.669698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:18.669090 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:17:18.879050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:18.878914 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:18.892541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:18.886105 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:19.024967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.017778 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:19.128736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.127955 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:19.308759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.302755 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:17:19.308759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.304896 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:17:19.348699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.347797 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:19.374731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.366960 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:17:19.375864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.375835 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:17:19.485945 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.485893 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:19.520653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.520601 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-8vgmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:17:19.672858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.672763 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:19.673100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.521861 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:17:19.696704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.663759 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-skqp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:17:19.785530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.777123 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-skqp6" podUID="f53ed47c-5e7c-4080-9452-63f49808674e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:19.785530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.778479 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:17:19.785530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.779646 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:17:19.879702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.879408 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:17:19.889759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.880663 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:17:19.889759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.887617 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:19.889759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:19.888156 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:17:20.056747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.051362 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:20.056747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.052849 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:17:20.155488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.141608 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:20.157181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.156074 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:17:20.174385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.169950 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded" Nov 02 22:17:20.196065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.196026 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:20.378835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.378792 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:17:20.576253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.576210 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:20.595697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.593550 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:17:20.659842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.652017 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:17:20.663166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.663136 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:17:20.666793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.666772 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:17:20.682948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.670875 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:17:20.726713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.718923 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:17:20.726713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.719257 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:17:20.927758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.924197 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:17:20.940642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.929965 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg"] Nov 02 22:17:20.952378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.945090 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:17:20.952378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:20.945140 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c759d5c-8xlgg"] Nov 02 22:17:22.212501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:22.212463 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d3d023-1380-4758-a6b5-d0c9098d1da0" path="/var/lib/kubelet/pods/83d3d023-1380-4758-a6b5-d0c9098d1da0/volumes" Nov 02 22:17:23.843788 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:23.843758 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:17:25.921519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:25.921481 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87"] Nov 02 22:17:25.930790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:25.930765 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:25.938648 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:25.936100 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87"] Nov 02 22:17:25.959030 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:25.959006 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2fc3ce-openshift-default-dockercfg-8vm4z\"" Nov 02 22:17:26.048725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.048693 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-istio-token\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.048886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.048776 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-envoy\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.048886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.048842 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-certs\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.048960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.048945 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-data\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.049002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.048976 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314c324-758d-4f87-985b-cab817bc6e31-istio-podinfo\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.049141 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.049118 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314c324-758d-4f87-985b-cab817bc6e31-istiod-ca-cert\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.049183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.049165 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-credential-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.049264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.049252 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.049325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.049313 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pf8s\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-kube-api-access-6pf8s\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151297 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-data\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151356 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314c324-758d-4f87-985b-cab817bc6e31-istio-podinfo\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151473 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314c324-758d-4f87-985b-cab817bc6e31-istiod-ca-cert\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151520 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-credential-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151592 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151710 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pf8s\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-kube-api-access-6pf8s\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151811 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151763 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-data\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151812 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-istio-token\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151870 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-envoy\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.151936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151917 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-certs\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.152055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.151937 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-credential-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.152055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.152029 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-socket\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.152199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.152181 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-certs\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.152285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.152263 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314c324-758d-4f87-985b-cab817bc6e31-istiod-ca-cert\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.154072 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.154049 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314c324-758d-4f87-985b-cab817bc6e31-istio-podinfo\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.154172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.154073 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-envoy\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.163761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.163739 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pf8s\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-kube-api-access-6pf8s\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.163858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.163785 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-istio-token\") pod \"e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.261264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.261179 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:26.624869 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:17:26.624837 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1314c324_758d_4f87_985b_cab817bc6e31.slice/crio-382459848d8109f5e11e5fe2d27a0eb54f4cff811d04e3c1519b98c778892281 WatchSource:0}: Error finding container 382459848d8109f5e11e5fe2d27a0eb54f4cff811d04e3c1519b98c778892281: Status 404 returned error can't find the container with id 382459848d8109f5e11e5fe2d27a0eb54f4cff811d04e3c1519b98c778892281 Nov 02 22:17:26.626970 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.626940 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87"] Nov 02 22:17:26.628293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.628258 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:26.628450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.628324 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:26.628450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.628356 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:17:26.838794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.838732 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" event={"ID":"1314c324-758d-4f87-985b-cab817bc6e31","Type":"ContainerStarted","Data":"34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd"} Nov 02 22:17:26.839026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.838822 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" event={"ID":"1314c324-758d-4f87-985b-cab817bc6e31","Type":"ContainerStarted","Data":"382459848d8109f5e11e5fe2d27a0eb54f4cff811d04e3c1519b98c778892281"} Nov 02 22:17:26.867802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:26.867738 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" podStartSLOduration=1.867720126 podStartE2EDuration="1.867720126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:26.860089037 +0000 UTC m=+1174.798202256" watchObservedRunningTime="2026-11-02 22:17:26.867720126 +0000 UTC m=+1174.805833346" Nov 02 22:17:27.272701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:27.268492 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:27.272701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:27.269482 4309 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 02 22:17:27.272701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:27.269564 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 02 22:17:28.262311 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:28.262271 4309 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 02 22:17:28.262487 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:28.262377 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 02 22:17:29.263779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:29.263739 4309 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 02 22:17:29.264310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:29.263812 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 02 22:17:30.262496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.262456 4309 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 02 22:17:30.262654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.262521 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 02 22:17:30.340981 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.340943 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh"] Nov 02 22:17:30.346947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.346915 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.351532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.351500 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:17:30.351745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.351644 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:17:30.351745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.351703 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:17:30.353597 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.353519 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh"] Nov 02 22:17:30.356310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.355894 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:17:30.379073 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.379045 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc2jf\" (UniqueName: \"kubernetes.io/projected/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-kube-api-access-zc2jf\") pod \"maas-api-revoke-sub-07063a45-zzvwh\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.379217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.379178 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07063a45-zzvwh\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.379455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.379416 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07063a45-zzvwh\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.480999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.480966 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07063a45-zzvwh\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.481178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.481018 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc2jf\" (UniqueName: \"kubernetes.io/projected/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-kube-api-access-zc2jf\") pod \"maas-api-revoke-sub-07063a45-zzvwh\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.481178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.481152 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07063a45-zzvwh\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.481608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.481583 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07063a45-zzvwh\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.523740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.521118 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07063a45-zzvwh\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.529915 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.529754 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc2jf\" (UniqueName: \"kubernetes.io/projected/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-kube-api-access-zc2jf\") pod \"maas-api-revoke-sub-07063a45-zzvwh\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:30.673144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:30.673110 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:31.066723 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:17:31.065753 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d92fa8_1047_4e44_b0d5_c0358f907f7f.slice/crio-3d2bf28b63efbbf49035829f1199ebcc55381dadbc13ea31b3ca038d36dfe556 WatchSource:0}: Error finding container 3d2bf28b63efbbf49035829f1199ebcc55381dadbc13ea31b3ca038d36dfe556: Status 404 returned error can't find the container with id 3d2bf28b63efbbf49035829f1199ebcc55381dadbc13ea31b3ca038d36dfe556 Nov 02 22:17:31.066723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:31.066610 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh"] Nov 02 22:17:31.262072 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:31.262044 4309 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 02 22:17:31.262235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:31.262088 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 02 22:17:31.942721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:31.941500 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" event={"ID":"d7d92fa8-1047-4e44-b0d5-c0358f907f7f","Type":"ContainerStarted","Data":"519649c0b371d1ac027e1b6e51f0fa340ac46eef4fcc21e1d4a89dee5ade3aaa"} Nov 02 22:17:31.942721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:31.941620 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" event={"ID":"d7d92fa8-1047-4e44-b0d5-c0358f907f7f","Type":"ContainerStarted","Data":"3d2bf28b63efbbf49035829f1199ebcc55381dadbc13ea31b3ca038d36dfe556"} Nov 02 22:17:31.960779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:31.960730 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" podStartSLOduration=1.9607140680000001 podStartE2EDuration="1.960714068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:31.959023919 +0000 UTC m=+1179.897137137" watchObservedRunningTime="2026-11-02 22:17:31.960714068 +0000 UTC m=+1179.898827286" Nov 02 22:17:32.262319 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:32.262242 4309 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 02 22:17:32.262319 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:32.262293 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 02 22:17:33.068702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:33.066453 4309 generic.go:356] "Generic (PLEG): container finished" podID="d7d92fa8-1047-4e44-b0d5-c0358f907f7f" containerID="519649c0b371d1ac027e1b6e51f0fa340ac46eef4fcc21e1d4a89dee5ade3aaa" exitCode=0 Nov 02 22:17:33.068702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:33.066733 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" event={"ID":"d7d92fa8-1047-4e44-b0d5-c0358f907f7f","Type":"ContainerDied","Data":"519649c0b371d1ac027e1b6e51f0fa340ac46eef4fcc21e1d4a89dee5ade3aaa"} Nov 02 22:17:33.084882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:33.079035 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb"] Nov 02 22:17:33.097636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:33.097588 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fd54212-2qfjb"] Nov 02 22:17:33.263239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:33.263203 4309 patch_prober.go:29] interesting pod/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 02 22:17:33.263571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:33.263542 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 02 22:17:34.450699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.438731 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:34.450699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.439864 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b0c9a5-c63e-4fb9-9272-3a210a448c9f" path="/var/lib/kubelet/pods/a6b0c9a5-c63e-4fb9-9272-3a210a448c9f/volumes" Nov 02 22:17:34.450699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.440507 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:34.450699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.441392 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:34.450699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.442613 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:17:34.540703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.536194 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-kube-api-access-zc2jf\" (UniqueName: \"kubernetes.io/projected/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-kube-api-access-zc2jf\") pod \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " Nov 02 22:17:34.540703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.536313 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-openshift-ca\") pod \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " Nov 02 22:17:34.540703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.536514 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-xks-ca\") pod \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\" (UID: \"d7d92fa8-1047-4e44-b0d5-c0358f907f7f\") " Nov 02 22:17:34.552222 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.548206 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-kube-api-access-zc2jf" pod "d7d92fa8-1047-4e44-b0d5-c0358f907f7f" (UID: "d7d92fa8-1047-4e44-b0d5-c0358f907f7f"). InnerVolumeSpecName "kube-api-access-zc2jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:34.570799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.566869 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-openshift-ca" pod "d7d92fa8-1047-4e44-b0d5-c0358f907f7f" (UID: "d7d92fa8-1047-4e44-b0d5-c0358f907f7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:34.605061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.595008 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-xks-ca" pod "d7d92fa8-1047-4e44-b0d5-c0358f907f7f" (UID: "d7d92fa8-1047-4e44-b0d5-c0358f907f7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:17:34.639614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.639575 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:34.639614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.639615 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc2jf\" (UniqueName: \"kubernetes.io/projected/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-kube-api-access-zc2jf\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:34.639943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:34.639632 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d92fa8-1047-4e44-b0d5-c0358f907f7f-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:35.089247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:35.089002 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" Nov 02 22:17:35.089247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:35.089063 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh" event={"ID":"d7d92fa8-1047-4e44-b0d5-c0358f907f7f","Type":"ContainerDied","Data":"3d2bf28b63efbbf49035829f1199ebcc55381dadbc13ea31b3ca038d36dfe556"} Nov 02 22:17:35.089515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:35.089458 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d2bf28b63efbbf49035829f1199ebcc55381dadbc13ea31b3ca038d36dfe556" Nov 02 22:17:36.036952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:36.036919 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz"] Nov 02 22:17:36.043764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:36.043734 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-067f2141-m5jqz"] Nov 02 22:17:36.192138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:36.192103 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2094c0-6717-4ecf-9b13-c3f6cb176a10" path="/var/lib/kubelet/pods/4e2094c0-6717-4ecf-9b13-c3f6cb176a10/volumes" Nov 02 22:17:37.407280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.407244 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk"] Nov 02 22:17:37.408188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.408169 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d7d92fa8-1047-4e44-b0d5-c0358f907f7f" containerName="revoke-keys" Nov 02 22:17:37.408252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.408191 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d92fa8-1047-4e44-b0d5-c0358f907f7f" containerName="revoke-keys" Nov 02 22:17:37.408445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.408427 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d92fa8-1047-4e44-b0d5-c0358f907f7f" containerName="revoke-keys" Nov 02 22:17:37.416305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.416282 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.419858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.419835 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk"] Nov 02 22:17:37.423721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.423700 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:17:37.423868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.423667 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:17:37.424845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.424826 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:17:37.424845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.424827 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:17:37.491556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.491521 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.491861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.491835 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.491944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.491895 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5xp\" (UniqueName: \"kubernetes.io/projected/d41c4c57-d428-4501-a361-fc51f6eaff1e-kube-api-access-lj5xp\") pod \"maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.593728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.593647 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.593947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.593846 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.593947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.593883 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5xp\" (UniqueName: \"kubernetes.io/projected/d41c4c57-d428-4501-a361-fc51f6eaff1e-kube-api-access-lj5xp\") pod \"maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.594484 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.594463 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.596128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.596106 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.603897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.603869 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5xp\" (UniqueName: \"kubernetes.io/projected/d41c4c57-d428-4501-a361-fc51f6eaff1e-kube-api-access-lj5xp\") pod \"maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:37.743949 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:37.743871 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:38.026002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:38.025929 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd"] Nov 02 22:17:38.034254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:38.034224 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407489b3-llrfd"] Nov 02 22:17:38.044707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:38.044652 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk"] Nov 02 22:17:38.045225 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:17:38.045201 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41c4c57_d428_4501_a361_fc51f6eaff1e.slice/crio-5dd5de8b060368248972f56d881b8732045fd25ad2558d8a88cd03f1fc0830eb WatchSource:0}: Error finding container 5dd5de8b060368248972f56d881b8732045fd25ad2558d8a88cd03f1fc0830eb: Status 404 returned error can't find the container with id 5dd5de8b060368248972f56d881b8732045fd25ad2558d8a88cd03f1fc0830eb Nov 02 22:17:38.148970 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:38.148919 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" event={"ID":"d41c4c57-d428-4501-a361-fc51f6eaff1e","Type":"ContainerStarted","Data":"5dd5de8b060368248972f56d881b8732045fd25ad2558d8a88cd03f1fc0830eb"} Nov 02 22:17:38.193612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:38.193572 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e943a27e-af4d-4644-a087-392e890a0120" path="/var/lib/kubelet/pods/e943a27e-af4d-4644-a087-392e890a0120/volumes" Nov 02 22:17:39.159864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:39.159823 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" event={"ID":"d41c4c57-d428-4501-a361-fc51f6eaff1e","Type":"ContainerStarted","Data":"44f74f21f7972a3ccde6c013af12c788e2c08fa6cf0467ac38e9a3c7f9709ecb"} Nov 02 22:17:39.178717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:39.178654 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" podStartSLOduration=2.178636914 podStartE2EDuration="2.178636914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:17:39.174237706 +0000 UTC m=+1187.112350925" watchObservedRunningTime="2026-11-02 22:17:39.178636914 +0000 UTC m=+1187.116750132" Nov 02 22:17:40.174121 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:40.174081 4309 generic.go:356] "Generic (PLEG): container finished" podID="d41c4c57-d428-4501-a361-fc51f6eaff1e" containerID="44f74f21f7972a3ccde6c013af12c788e2c08fa6cf0467ac38e9a3c7f9709ecb" exitCode=0 Nov 02 22:17:40.174599 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:40.174144 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" event={"ID":"d41c4c57-d428-4501-a361-fc51f6eaff1e","Type":"ContainerDied","Data":"44f74f21f7972a3ccde6c013af12c788e2c08fa6cf0467ac38e9a3c7f9709ecb"} Nov 02 22:17:41.534137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.534105 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:41.634246 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:17:41.634206 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:17:41.689168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.689127 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-openshift-ca\") pod \"d41c4c57-d428-4501-a361-fc51f6eaff1e\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " Nov 02 22:17:41.689424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.689367 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-xks-ca\") pod \"d41c4c57-d428-4501-a361-fc51f6eaff1e\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " Nov 02 22:17:41.689622 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.689561 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41c4c57-d428-4501-a361-fc51f6eaff1e-kube-api-access-lj5xp\" (UniqueName: \"kubernetes.io/projected/d41c4c57-d428-4501-a361-fc51f6eaff1e-kube-api-access-lj5xp\") pod \"d41c4c57-d428-4501-a361-fc51f6eaff1e\" (UID: \"d41c4c57-d428-4501-a361-fc51f6eaff1e\") " Nov 02 22:17:41.690017 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.689986 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-openshift-ca" pod "d41c4c57-d428-4501-a361-fc51f6eaff1e" (UID: "d41c4c57-d428-4501-a361-fc51f6eaff1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:41.690703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.690664 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:41.692633 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.692609 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-xks-ca" pod "d41c4c57-d428-4501-a361-fc51f6eaff1e" (UID: "d41c4c57-d428-4501-a361-fc51f6eaff1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:17:41.692844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.692809 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41c4c57-d428-4501-a361-fc51f6eaff1e-kube-api-access-lj5xp" pod "d41c4c57-d428-4501-a361-fc51f6eaff1e" (UID: "d41c4c57-d428-4501-a361-fc51f6eaff1e"). InnerVolumeSpecName "kube-api-access-lj5xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:41.792657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.792579 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj5xp\" (UniqueName: \"kubernetes.io/projected/d41c4c57-d428-4501-a361-fc51f6eaff1e-kube-api-access-lj5xp\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:41.792657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:41.792608 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d41c4c57-d428-4501-a361-fc51f6eaff1e-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:42.354725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:42.352806 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" event={"ID":"d41c4c57-d428-4501-a361-fc51f6eaff1e","Type":"ContainerDied","Data":"5dd5de8b060368248972f56d881b8732045fd25ad2558d8a88cd03f1fc0830eb"} Nov 02 22:17:42.354725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:42.352883 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk" Nov 02 22:17:42.354725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:42.352886 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dd5de8b060368248972f56d881b8732045fd25ad2558d8a88cd03f1fc0830eb" Nov 02 22:17:42.441144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:42.441105 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh"] Nov 02 22:17:42.448069 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:42.448025 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-fbpnh"] Nov 02 22:17:44.192501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:44.192462 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd" path="/var/lib/kubelet/pods/b6eb61f7-d2c8-4eb4-8dd5-c14f37e5a5dd/volumes" Nov 02 22:17:46.610484 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.610449 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz"] Nov 02 22:17:46.611483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.611459 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d41c4c57-d428-4501-a361-fc51f6eaff1e" containerName="revoke-keys" Nov 02 22:17:46.611483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.611484 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41c4c57-d428-4501-a361-fc51f6eaff1e" containerName="revoke-keys" Nov 02 22:17:46.611738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.611723 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41c4c57-d428-4501-a361-fc51f6eaff1e" containerName="revoke-keys" Nov 02 22:17:46.617995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.617968 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.623728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.623702 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz"] Nov 02 22:17:46.626196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.626175 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:17:46.626342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.626320 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:17:46.626417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.626407 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:17:46.626499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.626485 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:17:46.719057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.719022 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b9d0683-fvjcz\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.719257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.719237 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv58d\" (UniqueName: \"kubernetes.io/projected/c7f3196e-3c3f-4748-8789-bc2147772425-kube-api-access-fv58d\") pod \"maas-api-revoke-sub-1b9d0683-fvjcz\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.719410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.719392 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b9d0683-fvjcz\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.820972 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.820864 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b9d0683-fvjcz\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.821216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.821095 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv58d\" (UniqueName: \"kubernetes.io/projected/c7f3196e-3c3f-4748-8789-bc2147772425-kube-api-access-fv58d\") pod \"maas-api-revoke-sub-1b9d0683-fvjcz\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.821306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.821244 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b9d0683-fvjcz\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.823863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.823803 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b9d0683-fvjcz\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.826516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.826484 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b9d0683-fvjcz\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.835145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.835080 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv58d\" (UniqueName: \"kubernetes.io/projected/c7f3196e-3c3f-4748-8789-bc2147772425-kube-api-access-fv58d\") pod \"maas-api-revoke-sub-1b9d0683-fvjcz\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:46.940738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:46.940643 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:47.234489 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:17:47.234452 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f3196e_3c3f_4748_8789_bc2147772425.slice/crio-2110df20459d7a278520e658e44f14ebd020664f401753d1927095add90e8004 WatchSource:0}: Error finding container 2110df20459d7a278520e658e44f14ebd020664f401753d1927095add90e8004: Status 404 returned error can't find the container with id 2110df20459d7a278520e658e44f14ebd020664f401753d1927095add90e8004 Nov 02 22:17:47.234762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:47.234737 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz"] Nov 02 22:17:47.422815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:47.422774 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" event={"ID":"c7f3196e-3c3f-4748-8789-bc2147772425","Type":"ContainerStarted","Data":"2110df20459d7a278520e658e44f14ebd020664f401753d1927095add90e8004"} Nov 02 22:17:47.561704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:47.561652 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l"] Nov 02 22:17:48.441297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:48.441263 4309 generic.go:356] "Generic (PLEG): container finished" podID="c7f3196e-3c3f-4748-8789-bc2147772425" containerID="0a73ee3f9f62f69c4761b19d96f5137c39507b1c6c1fda33b8ad13c3840b4d50" exitCode=0 Nov 02 22:17:48.441815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:48.441333 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" event={"ID":"c7f3196e-3c3f-4748-8789-bc2147772425","Type":"ContainerDied","Data":"0a73ee3f9f62f69c4761b19d96f5137c39507b1c6c1fda33b8ad13c3840b4d50"} Nov 02 22:17:49.758785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.758753 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:49.839159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.839126 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f3196e-3c3f-4748-8789-bc2147772425-kube-api-access-fv58d\" (UniqueName: \"kubernetes.io/projected/c7f3196e-3c3f-4748-8789-bc2147772425-kube-api-access-fv58d\") pod \"c7f3196e-3c3f-4748-8789-bc2147772425\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " Nov 02 22:17:49.839159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.839171 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-openshift-ca\") pod \"c7f3196e-3c3f-4748-8789-bc2147772425\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " Nov 02 22:17:49.839422 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.839350 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-xks-ca\") pod \"c7f3196e-3c3f-4748-8789-bc2147772425\" (UID: \"c7f3196e-3c3f-4748-8789-bc2147772425\") " Nov 02 22:17:49.839626 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.839595 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-openshift-ca" pod "c7f3196e-3c3f-4748-8789-bc2147772425" (UID: "c7f3196e-3c3f-4748-8789-bc2147772425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:49.840284 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.840257 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:49.842316 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.842289 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f3196e-3c3f-4748-8789-bc2147772425-kube-api-access-fv58d" pod "c7f3196e-3c3f-4748-8789-bc2147772425" (UID: "c7f3196e-3c3f-4748-8789-bc2147772425"). InnerVolumeSpecName "kube-api-access-fv58d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:49.842590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.842565 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-xks-ca" pod "c7f3196e-3c3f-4748-8789-bc2147772425" (UID: "c7f3196e-3c3f-4748-8789-bc2147772425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:17:49.942553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.942516 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv58d\" (UniqueName: \"kubernetes.io/projected/c7f3196e-3c3f-4748-8789-bc2147772425-kube-api-access-fv58d\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:49.942553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:49.942550 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7f3196e-3c3f-4748-8789-bc2147772425-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:50.474739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:50.473137 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" Nov 02 22:17:50.474739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:50.473204 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz" event={"ID":"c7f3196e-3c3f-4748-8789-bc2147772425","Type":"ContainerDied","Data":"2110df20459d7a278520e658e44f14ebd020664f401753d1927095add90e8004"} Nov 02 22:17:50.474739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:50.473250 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2110df20459d7a278520e658e44f14ebd020664f401753d1927095add90e8004" Nov 02 22:17:52.028537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:52.028495 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd"] Nov 02 22:17:52.037410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:52.037377 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1e8d2b-vc9cd"] Nov 02 22:17:52.194327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:52.194293 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c986e4-40d6-4841-9e86-cef998f99160" path="/var/lib/kubelet/pods/a8c986e4-40d6-4841-9e86-cef998f99160/volumes" Nov 02 22:17:52.577002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:52.576958 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" podUID="3f2411b0-3bee-45dc-8ac7-c9acffe89af0" containerName="payload-processing" containerID="cri-o://3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408" gracePeriod=25 Nov 02 22:17:52.914734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:52.914699 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:17:53.071668 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.071628 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-plugins-config-volume\") pod \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\" (UID: \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\") " Nov 02 22:17:53.072230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.072041 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-kube-api-access-rl2rh\" (UniqueName: \"kubernetes.io/projected/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-kube-api-access-rl2rh\") pod \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\" (UID: \"3f2411b0-3bee-45dc-8ac7-c9acffe89af0\") " Nov 02 22:17:53.073262 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.073231 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-plugins-config-volume" pod "3f2411b0-3bee-45dc-8ac7-c9acffe89af0" (UID: "3f2411b0-3bee-45dc-8ac7-c9acffe89af0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:17:53.081754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.081645 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-kube-api-access-rl2rh" pod "3f2411b0-3bee-45dc-8ac7-c9acffe89af0" (UID: "3f2411b0-3bee-45dc-8ac7-c9acffe89af0"). InnerVolumeSpecName "kube-api-access-rl2rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:17:53.175947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.175829 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2rh\" (UniqueName: \"kubernetes.io/projected/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-kube-api-access-rl2rh\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:53.175947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.175873 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2411b0-3bee-45dc-8ac7-c9acffe89af0-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:17:53.555763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.555072 4309 generic.go:356] "Generic (PLEG): container finished" podID="3f2411b0-3bee-45dc-8ac7-c9acffe89af0" containerID="3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408" exitCode=0 Nov 02 22:17:53.555763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.555157 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" event={"ID":"3f2411b0-3bee-45dc-8ac7-c9acffe89af0","Type":"ContainerDied","Data":"3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408"} Nov 02 22:17:53.555763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.555205 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" event={"ID":"3f2411b0-3bee-45dc-8ac7-c9acffe89af0","Type":"ContainerDied","Data":"e5513f3f53b7b7c0cb1e172e655b2bcec09a6229b726edb20e20746a1e65288d"} Nov 02 22:17:53.555763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.555236 4309 scope.go:122] "RemoveContainer" containerID="3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408" Nov 02 22:17:53.555763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.555402 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l" Nov 02 22:17:53.607745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.607335 4309 scope.go:122] "RemoveContainer" containerID="3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408" Nov 02 22:17:53.608305 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:17:53.608273 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408\": container with ID starting with 3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408 not found: ID does not exist" containerID="3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408" Nov 02 22:17:53.608412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.608330 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408"} err="failed to get container status \"3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408\": rpc error: code = NotFound desc = could not find container \"3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408\": container with ID starting with 3a893cfaf33d171f448b16d162b97fa5ed4ff5308d50ff5316584f57e04cf408 not found: ID does not exist" Nov 02 22:17:53.623731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.621774 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l"] Nov 02 22:17:53.625182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.625153 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2fc3ce-7c4f78c5c5-s6d2l"] Nov 02 22:17:53.979945 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:53.979904 4309 scope.go:122] "RemoveContainer" containerID="1ef600122737efda3914947957b4669d2edcb3bf1fdaac4a97080cb29954a576" Nov 02 22:17:54.009437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:54.009407 4309 scope.go:122] "RemoveContainer" containerID="098cda27f9bacc98522a50a2dd778451520190993b8d7f0abf0fbca1a109528b" Nov 02 22:17:54.132784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:54.132740 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr"] Nov 02 22:17:54.174205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:54.174039 4309 scope.go:122] "RemoveContainer" containerID="0aafa0595c74481c099829af12ddf4caa487df000c25de072a2aa81932cdbfc7" Nov 02 22:17:54.284857 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:54.284822 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2411b0-3bee-45dc-8ac7-c9acffe89af0" path="/var/lib/kubelet/pods/3f2411b0-3bee-45dc-8ac7-c9acffe89af0/volumes" Nov 02 22:17:54.287516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:54.287491 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e3a1ba2-4zlvr"] Nov 02 22:17:54.319383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:54.319356 4309 scope.go:122] "RemoveContainer" containerID="88097bdafedd8b996b829a16f9679ab4f7f5445afe8b3b763bcb2dfb02c97b44" Nov 02 22:17:54.339954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:54.339926 4309 scope.go:122] "RemoveContainer" containerID="0a252d0b7f457ea9a7dda2242e7cadaf8bebd870139ba426e8597c7bc2a1759c" Nov 02 22:17:55.739701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:55.739190 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:17:55.893697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:55.892755 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:55.931748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:55.929010 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:56.289699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:56.288943 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:17:56.310698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:56.304713 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c45e5db9-eea7-43cf-ba46-1595d9511182" path="/var/lib/kubelet/pods/c45e5db9-eea7-43cf-ba46-1595d9511182/volumes" Nov 02 22:17:56.310698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:56.307312 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:17:56.377699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:56.375781 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:56.461822 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:56.461765 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:57.005755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.005710 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded" Nov 02 22:17:57.276796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.272975 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:57.374092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.351918 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:57.477697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.473122 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:57.477697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.474585 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:17:57.522118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.522079 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:17:57.523336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.523304 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:57.688756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.687798 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:57.744695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.739827 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:57.789697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.787839 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:57.945765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:57.929613 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:17:58.008243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.007562 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:17:58.351700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.348232 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:17:58.388699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.382296 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:17:58.388699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.382936 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:58.491901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.491857 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:58.549710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.538699 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:58.549710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.542708 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 22:17:58.624421 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.624328 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 02 22:17:58.688298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.688256 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:58.705961 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.705928 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:17:58.706529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.706509 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:17:58.707833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.707816 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:17:58.719806 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.719780 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:17:58.720021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.720004 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:17:58.740034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.739995 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:58.778276 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.778241 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:17:58.797741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.797699 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:58.800220 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.800195 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:17:58.941934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.941833 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:17:58.960705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:58.943023 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:17:59.052329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.014962 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:17:59.074734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.054612 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:17:59.138151 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.138108 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:59.383473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.383423 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:17:59.437302 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.437233 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:59.717753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.438252 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:17:59.717753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.439075 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:17:59.717753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.554499 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:59.717753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.554917 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:59.717753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.556452 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:17:59.717753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.697219 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:17:59.778139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.770030 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:17:59.832704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:17:59.826907 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:00.042235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.042129 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:18:00.115622 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.115576 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded" Nov 02 22:18:00.218670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.218632 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:00.419412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.417317 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:00.454831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.454764 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": context deadline exceeded" Nov 02 22:18:00.456496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.455861 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:18:00.528721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.510774 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:18:00.561570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.559351 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:00.561570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.561045 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:00.573759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.561930 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:18:00.582723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.580367 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:18:00.699064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.698963 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:00.832722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.805109 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 02 22:18:00.885755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.868781 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:00.907922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.907891 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:18:00.914853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.914829 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:18:00.957519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.957432 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:18:00.958807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.958774 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:18:00.962201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.962063 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:18:00.984739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:00.979790 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:18:01.084700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:01.078500 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:18:01.122531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:01.122501 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:18:01.219870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:01.219785 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:18:03.643853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.643809 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g"] Nov 02 22:18:03.645160 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.645136 4309 cpu_manager.go:414] "containerMap: removing container" podUID="c7f3196e-3c3f-4748-8789-bc2147772425" containerName="revoke-keys" Nov 02 22:18:03.645212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.645164 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f3196e-3c3f-4748-8789-bc2147772425" containerName="revoke-keys" Nov 02 22:18:03.645255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.645222 4309 cpu_manager.go:414] "containerMap: removing container" podUID="3f2411b0-3bee-45dc-8ac7-c9acffe89af0" containerName="payload-processing" Nov 02 22:18:03.645255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.645230 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2411b0-3bee-45dc-8ac7-c9acffe89af0" containerName="payload-processing" Nov 02 22:18:03.645439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.645423 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2411b0-3bee-45dc-8ac7-c9acffe89af0" containerName="payload-processing" Nov 02 22:18:03.645479 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.645462 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f3196e-3c3f-4748-8789-bc2147772425" containerName="revoke-keys" Nov 02 22:18:03.664976 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.661064 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g"] Nov 02 22:18:03.664976 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.661189 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:03.689711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.689668 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14dae2-dockercfg-n97hr\"" Nov 02 22:18:03.753700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.743867 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14dae2\"" Nov 02 22:18:03.775700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.772059 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtqmm\" (UniqueName: \"kubernetes.io/projected/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-kube-api-access-jtqmm\") pod \"payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g\" (UID: \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:03.775700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.772204 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g\" (UID: \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:03.877168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.877125 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqmm\" (UniqueName: \"kubernetes.io/projected/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-kube-api-access-jtqmm\") pod \"payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g\" (UID: \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:03.879024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.878996 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g\" (UID: \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:03.899955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.899878 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g\" (UID: \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:03.952724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:03.933171 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqmm\" (UniqueName: \"kubernetes.io/projected/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-kube-api-access-jtqmm\") pod \"payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g\" (UID: \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:04.029699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:04.028951 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:04.458669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:04.458630 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g"] Nov 02 22:18:04.469705 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:18:04.469585 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b051708_5a84_4f0f_ba1a_b6ea2ff48099.slice/crio-a7a2935694e49793a4381ff6c1986d4b83870862591979c22ae9b70050525e9c WatchSource:0}: Error finding container a7a2935694e49793a4381ff6c1986d4b83870862591979c22ae9b70050525e9c: Status 404 returned error can't find the container with id a7a2935694e49793a4381ff6c1986d4b83870862591979c22ae9b70050525e9c Nov 02 22:18:05.113595 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:05.112726 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" event={"ID":"8b051708-5a84-4f0f-ba1a-b6ea2ff48099","Type":"ContainerStarted","Data":"89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d"} Nov 02 22:18:05.113595 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:05.112793 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" event={"ID":"8b051708-5a84-4f0f-ba1a-b6ea2ff48099","Type":"ContainerStarted","Data":"a7a2935694e49793a4381ff6c1986d4b83870862591979c22ae9b70050525e9c"} Nov 02 22:18:05.113595 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:05.112822 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:05.136266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:05.135767 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" podStartSLOduration=2.135748142 podStartE2EDuration="2.135748142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:05.129295476 +0000 UTC m=+1213.067408725" watchObservedRunningTime="2026-11-02 22:18:05.135748142 +0000 UTC m=+1213.073861361" Nov 02 22:18:05.418820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:05.418731 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk"] Nov 02 22:18:05.433182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:05.425901 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2fc3ce-2rdjk"] Nov 02 22:18:06.219244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:06.219210 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41c4c57-d428-4501-a361-fc51f6eaff1e" path="/var/lib/kubelet/pods/d41c4c57-d428-4501-a361-fc51f6eaff1e/volumes" Nov 02 22:18:07.782769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.782684 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj"] Nov 02 22:18:07.787825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.787800 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.793952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.793926 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-84a94060-openshift-default-dockercfg-kccrv\"" Nov 02 22:18:07.799462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.799439 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj"] Nov 02 22:18:07.869274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.869180 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-podinfo\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.869274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.869232 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d6666a6-9da1-4eca-9b73-2872991b77cc-istiod-ca-cert\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.869516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.869401 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-envoy\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.869589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.869571 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.869660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.869630 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-credential-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.869771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.869755 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-certs\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.869912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.869893 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5w6\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-kube-api-access-jl5w6\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.870100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.870078 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-data\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.870150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.870128 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-token\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.972082 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.972252 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-credential-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.972357 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-certs\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.972512 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5w6\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-kube-api-access-jl5w6\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.972663 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-data\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.972719 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-token\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.972777 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-podinfo\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.972801 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d6666a6-9da1-4eca-9b73-2872991b77cc-istiod-ca-cert\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.972869 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-envoy\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.973927 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-data\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.974864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.974463 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.975741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.975254 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d6666a6-9da1-4eca-9b73-2872991b77cc-istiod-ca-cert\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.975741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.975412 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-credential-socket\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.975741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.975720 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-certs\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.978129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.978105 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-envoy\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.978244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.978186 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-podinfo\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.981088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.981063 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5w6\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-kube-api-access-jl5w6\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:07.981221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:07.981199 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-token\") pod \"e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:08.113766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.113713 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:08.155832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.155793 4309 generic.go:356] "Generic (PLEG): container finished" podID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerID="213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29" exitCode=0 Nov 02 22:18:08.156011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.155832 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" event={"ID":"c0762470-8657-4a0e-b29f-e6064a65d2ca","Type":"ContainerDied","Data":"213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29"} Nov 02 22:18:08.156465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.156441 4309 scope.go:122] "RemoveContainer" containerID="213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29" Nov 02 22:18:08.158936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.158900 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:18:08.159057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.158995 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:18:08.159057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.159042 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:18:08.431847 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:18:08.431812 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6666a6_9da1_4eca_9b73_2872991b77cc.slice/crio-7804a6c8328991dd4307c0bfd3519fda892bc63f9c1688981a8791587c170388 WatchSource:0}: Error finding container 7804a6c8328991dd4307c0bfd3519fda892bc63f9c1688981a8791587c170388: Status 404 returned error can't find the container with id 7804a6c8328991dd4307c0bfd3519fda892bc63f9c1688981a8791587c170388 Nov 02 22:18:08.432512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.432492 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj"] Nov 02 22:18:08.435431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.435389 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:18:08.435531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.435482 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:18:08.435577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.435525 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:18:08.721815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:08.719843 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:18:09.078722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:09.073053 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2"] Nov 02 22:18:09.120759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:09.107958 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f0c6d19-m46m2"] Nov 02 22:18:09.263698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:09.263605 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" event={"ID":"3d6666a6-9da1-4eca-9b73-2872991b77cc","Type":"ContainerStarted","Data":"354ca6fccc191fa28de211107c89c89ec522b9386ce90fdf8f5aa6fd55896571"} Nov 02 22:18:09.263698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:09.263692 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" event={"ID":"3d6666a6-9da1-4eca-9b73-2872991b77cc","Type":"ContainerStarted","Data":"7804a6c8328991dd4307c0bfd3519fda892bc63f9c1688981a8791587c170388"} Nov 02 22:18:09.289699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:09.283909 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" event={"ID":"c0762470-8657-4a0e-b29f-e6064a65d2ca","Type":"ContainerStarted","Data":"c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925"} Nov 02 22:18:09.307706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:09.307113 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" podStartSLOduration=2.307090745 podStartE2EDuration="2.307090745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:09.290091079 +0000 UTC m=+1217.228204297" watchObservedRunningTime="2026-11-02 22:18:09.307090745 +0000 UTC m=+1217.245203964" Nov 02 22:18:09.343754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:09.325370 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:18:09.343754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:09.325515 4309 patch_prober.go:29] interesting pod/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:18:09.343754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:09.325566 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 22:18:10.049733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.046977 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94"] Nov 02 22:18:10.056478 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.056388 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c7de6f-4lw94"] Nov 02 22:18:10.062604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.062580 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn"] Nov 02 22:18:10.068569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.068547 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc3d4caa-cr9xn"] Nov 02 22:18:10.115699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.115602 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:10.116207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.115795 4309 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:18:10.116207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.115842 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:18:10.249177 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.249137 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4" path="/var/lib/kubelet/pods/13e8c7e9-d07a-43d5-b8ba-9b82ec7b9fd4/volumes" Nov 02 22:18:10.250159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.250140 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7100abcf-75ec-4990-9ead-fc45594e86d2" path="/var/lib/kubelet/pods/7100abcf-75ec-4990-9ead-fc45594e86d2/volumes" Nov 02 22:18:10.250870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.250854 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84cd6da1-1462-443b-b0f8-957a1073a6ea" path="/var/lib/kubelet/pods/84cd6da1-1462-443b-b0f8-957a1073a6ea/volumes" Nov 02 22:18:10.323814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.323240 4309 patch_prober.go:29] interesting pod/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:18:10.323814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:10.323303 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 22:18:11.054697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:11.054640 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7"] Nov 02 22:18:11.061617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:11.061590 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b895d99d-p5xd7"] Nov 02 22:18:11.117877 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:11.117844 4309 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:18:11.118464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:11.118432 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:18:11.334823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:11.330733 4309 patch_prober.go:29] interesting pod/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:18:11.334823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:11.330792 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 22:18:11.647554 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:18:11.647452 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:18:12.116229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:12.116193 4309 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:18:12.116527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:12.116478 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:18:12.249778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:12.249243 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9192c071-8968-4340-9e0b-65231b50255f" path="/var/lib/kubelet/pods/9192c071-8968-4340-9e0b-65231b50255f/volumes" Nov 02 22:18:12.317714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:12.317662 4309 patch_prober.go:29] interesting pod/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:18:12.317998 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:12.317970 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 22:18:12.462426 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:12.462260 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87"] Nov 02 22:18:12.462613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:12.462472 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" containerID="cri-o://34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd" gracePeriod=30 Nov 02 22:18:13.158858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.158820 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk"] Nov 02 22:18:13.193700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.181014 4309 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:18:13.193700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.181100 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:18:13.275392 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.275358 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.280955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.279975 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk"] Nov 02 22:18:13.280955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.279991 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:18:13.280955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.280253 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:18:13.280955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.280418 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:18:13.280955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.280333 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:18:13.347982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.347946 4309 patch_prober.go:29] interesting pod/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:18:13.348267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.348237 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 22:18:13.351925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.351903 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj"] Nov 02 22:18:13.360747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.360724 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee61efa0-728sj"] Nov 02 22:18:13.380746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.380718 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0dd701-btmqk\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.381043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.381018 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szxp\" (UniqueName: \"kubernetes.io/projected/3529b590-0d55-4064-87aa-7a02e242e454-kube-api-access-7szxp\") pod \"maas-api-revoke-keys-e2e-api-b-0dd701-btmqk\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.381142 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.381107 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0dd701-btmqk\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.494014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.493914 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0dd701-btmqk\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.515707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.495602 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szxp\" (UniqueName: \"kubernetes.io/projected/3529b590-0d55-4064-87aa-7a02e242e454-kube-api-access-7szxp\") pod \"maas-api-revoke-keys-e2e-api-b-0dd701-btmqk\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.517568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.517526 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0dd701-btmqk\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.541053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.494999 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0dd701-btmqk\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.545529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.545453 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0dd701-btmqk\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.547802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.547341 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szxp\" (UniqueName: \"kubernetes.io/projected/3529b590-0d55-4064-87aa-7a02e242e454-kube-api-access-7szxp\") pod \"maas-api-revoke-keys-e2e-api-b-0dd701-btmqk\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:13.672738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:13.670414 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:14.134053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:14.133958 4309 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:18:14.134336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:14.134306 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:18:14.325289 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:14.324818 4309 patch_prober.go:29] interesting pod/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:18:14.325289 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:14.324880 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 22:18:14.337420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:14.337388 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b0299b-2bde-4fa6-a626-81264f95ea04" path="/var/lib/kubelet/pods/63b0299b-2bde-4fa6-a626-81264f95ea04/volumes" Nov 02 22:18:14.338194 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:14.338173 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk"] Nov 02 22:18:14.474750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:14.474206 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" event={"ID":"3529b590-0d55-4064-87aa-7a02e242e454","Type":"ContainerStarted","Data":"9c7f5cb8056ebd768a61732c0929fbe84b50795040ea81308e81a6ec7254b858"} Nov 02 22:18:15.127865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:15.127759 4309 patch_prober.go:29] interesting pod/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 02 22:18:15.128138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:15.128106 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 02 22:18:15.318090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:15.318052 4309 patch_prober.go:29] interesting pod/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 22:18:15.318334 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:15.318308 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 22:18:15.524049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:15.524006 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" event={"ID":"3529b590-0d55-4064-87aa-7a02e242e454","Type":"ContainerStarted","Data":"9d2a2dc6b1379a1d274ce79546a1a03ca2158c848300d1f4751c40a5fed8a8af"} Nov 02 22:18:15.546973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:15.546912 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" podStartSLOduration=3.546893329 podStartE2EDuration="3.546893329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:15.539168341 +0000 UTC m=+1223.477281560" watchObservedRunningTime="2026-11-02 22:18:15.546893329 +0000 UTC m=+1223.485006548" Nov 02 22:18:16.131010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:16.130981 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:18:16.294061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:16.294032 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:16.295084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:16.294875 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:16.295499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:16.295479 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:18:16.326889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:16.326837 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:18:16.327603 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:16.327556 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:18:16.334735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:16.328995 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:18:16.558108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:16.558072 4309 generic.go:356] "Generic (PLEG): container finished" podID="3529b590-0d55-4064-87aa-7a02e242e454" containerID="9d2a2dc6b1379a1d274ce79546a1a03ca2158c848300d1f4751c40a5fed8a8af" exitCode=0 Nov 02 22:18:16.558513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:16.558147 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" event={"ID":"3529b590-0d55-4064-87aa-7a02e242e454","Type":"ContainerDied","Data":"9d2a2dc6b1379a1d274ce79546a1a03ca2158c848300d1f4751c40a5fed8a8af"} Nov 02 22:18:17.634469 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.633425 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549"] Nov 02 22:18:17.640735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.640644 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549"] Nov 02 22:18:17.640909 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.640779 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:17.688050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.688020 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-df8835-openshift-default-dockercfg-h6twj\"" Nov 02 22:18:17.964926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.922859 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-token\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:17.964926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.922921 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-credential-socket\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:17.964926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.923007 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-socket\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:17.964926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.923118 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-envoy\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:17.964926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.923231 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-podinfo\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:17.964926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.923278 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmqh\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-kube-api-access-jwmqh\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:17.964926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.923357 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istiod-ca-cert\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:17.964926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.923531 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-certs\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:17.964926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:17.923566 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-data\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.044662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.044558 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-token\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.044851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.044712 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-credential-socket\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.044922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.044841 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-socket\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.045033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.045013 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-envoy\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.045196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.045177 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-podinfo\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.045269 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.045246 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmqh\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-kube-api-access-jwmqh\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.045386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.045360 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istiod-ca-cert\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.045546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.045515 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-certs\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.045624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.045574 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-data\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.046376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.046333 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-data\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.047101 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.047061 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-credential-socket\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.047428 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.047405 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-socket\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.050899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.050858 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-envoy\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.053697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.053646 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-podinfo\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.054616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.054579 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istiod-ca-cert\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.054838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.054813 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-certs\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.082007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.078425 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmqh\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-kube-api-access-jwmqh\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.088735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.088689 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-token\") pod \"e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.179910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.179879 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:18:18.184033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.183974 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:18.254696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.254646 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-data\") pod \"1314c324-758d-4f87-985b-cab817bc6e31\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " Nov 02 22:18:18.255020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.255000 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-envoy\") pod \"1314c324-758d-4f87-985b-cab817bc6e31\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " Nov 02 22:18:18.255808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.255787 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-openshift-ca\") pod \"3529b590-0d55-4064-87aa-7a02e242e454\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " Nov 02 22:18:18.255952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.255930 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-certs\") pod \"1314c324-758d-4f87-985b-cab817bc6e31\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " Nov 02 22:18:18.256086 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.256006 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-xks-ca\") pod \"3529b590-0d55-4064-87aa-7a02e242e454\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " Nov 02 22:18:18.256330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.256310 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-istio-token\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-istio-token\") pod \"1314c324-758d-4f87-985b-cab817bc6e31\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " Nov 02 22:18:18.256433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.256379 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-kube-api-access-6pf8s\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-kube-api-access-6pf8s\") pod \"1314c324-758d-4f87-985b-cab817bc6e31\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " Nov 02 22:18:18.256433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.256412 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-credential-socket\") pod \"1314c324-758d-4f87-985b-cab817bc6e31\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " Nov 02 22:18:18.256559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.256455 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1314c324-758d-4f87-985b-cab817bc6e31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314c324-758d-4f87-985b-cab817bc6e31-istiod-ca-cert\") pod \"1314c324-758d-4f87-985b-cab817bc6e31\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " Nov 02 22:18:18.256559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.256476 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-socket\") pod \"1314c324-758d-4f87-985b-cab817bc6e31\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " Nov 02 22:18:18.256559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.256506 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3529b590-0d55-4064-87aa-7a02e242e454-kube-api-access-7szxp\" (UniqueName: \"kubernetes.io/projected/3529b590-0d55-4064-87aa-7a02e242e454-kube-api-access-7szxp\") pod \"3529b590-0d55-4064-87aa-7a02e242e454\" (UID: \"3529b590-0d55-4064-87aa-7a02e242e454\") " Nov 02 22:18:18.256559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.256537 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1314c324-758d-4f87-985b-cab817bc6e31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314c324-758d-4f87-985b-cab817bc6e31-istio-podinfo\") pod \"1314c324-758d-4f87-985b-cab817bc6e31\" (UID: \"1314c324-758d-4f87-985b-cab817bc6e31\") " Nov 02 22:18:18.258444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.258341 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-data" pod "1314c324-758d-4f87-985b-cab817bc6e31" (UID: "1314c324-758d-4f87-985b-cab817bc6e31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:18.259323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.259236 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-openshift-ca" pod "3529b590-0d55-4064-87aa-7a02e242e454" (UID: "3529b590-0d55-4064-87aa-7a02e242e454"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:18.259444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.259310 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-certs" pod "1314c324-758d-4f87-985b-cab817bc6e31" (UID: "1314c324-758d-4f87-985b-cab817bc6e31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:18.259897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.259870 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1314c324-758d-4f87-985b-cab817bc6e31-istiod-ca-cert" pod "1314c324-758d-4f87-985b-cab817bc6e31" (UID: "1314c324-758d-4f87-985b-cab817bc6e31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:18.260210 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.260185 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-credential-socket" pod "1314c324-758d-4f87-985b-cab817bc6e31" (UID: "1314c324-758d-4f87-985b-cab817bc6e31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:18.260659 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.260639 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-socket" pod "1314c324-758d-4f87-985b-cab817bc6e31" (UID: "1314c324-758d-4f87-985b-cab817bc6e31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:18.267891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.267594 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:18.272707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.270372 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1314c324-758d-4f87-985b-cab817bc6e31-istio-podinfo" pod "1314c324-758d-4f87-985b-cab817bc6e31" (UID: "1314c324-758d-4f87-985b-cab817bc6e31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:18:18.281721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.278206 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-xks-ca" pod "3529b590-0d55-4064-87aa-7a02e242e454" (UID: "3529b590-0d55-4064-87aa-7a02e242e454"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:18.281721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.278414 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-kube-api-access-6pf8s" pod "1314c324-758d-4f87-985b-cab817bc6e31" (UID: "1314c324-758d-4f87-985b-cab817bc6e31"). InnerVolumeSpecName "kube-api-access-6pf8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:18.281721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.280026 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-envoy" pod "1314c324-758d-4f87-985b-cab817bc6e31" (UID: "1314c324-758d-4f87-985b-cab817bc6e31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:18.281721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.280136 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3529b590-0d55-4064-87aa-7a02e242e454-kube-api-access-7szxp" pod "3529b590-0d55-4064-87aa-7a02e242e454" (UID: "3529b590-0d55-4064-87aa-7a02e242e454"). InnerVolumeSpecName "kube-api-access-7szxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:18.281721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.280285 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-istio-token" pod "1314c324-758d-4f87-985b-cab817bc6e31" (UID: "1314c324-758d-4f87-985b-cab817bc6e31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:18.360467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360431 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pf8s\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-kube-api-access-6pf8s\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360578 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360603 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1314c324-758d-4f87-985b-cab817bc6e31-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360620 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360636 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szxp\" (UniqueName: \"kubernetes.io/projected/3529b590-0d55-4064-87aa-7a02e242e454-kube-api-access-7szxp\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360651 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1314c324-758d-4f87-985b-cab817bc6e31-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360666 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360729 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360742 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360756 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1314c324-758d-4f87-985b-cab817bc6e31-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360769 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3529b590-0d55-4064-87aa-7a02e242e454-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.360782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.360782 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1314c324-758d-4f87-985b-cab817bc6e31-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.606213 4309 generic.go:356] "Generic (PLEG): container finished" podID="1314c324-758d-4f87-985b-cab817bc6e31" containerID="34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd" exitCode=0 Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.606332 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" event={"ID":"1314c324-758d-4f87-985b-cab817bc6e31","Type":"ContainerDied","Data":"34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd"} Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.606386 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" event={"ID":"1314c324-758d-4f87-985b-cab817bc6e31","Type":"ContainerDied","Data":"382459848d8109f5e11e5fe2d27a0eb54f4cff811d04e3c1519b98c778892281"} Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.606415 4309 scope.go:122] "RemoveContainer" containerID="34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd" Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.606561 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549"] Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.606587 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87" Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.606780 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.606857 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.606900 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:18:18.609750 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:18:18.607077 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5748639_a17c_4ea1_9400_ac7fd3519cc7.slice/crio-76792d9ca4104709c4f2e78c6c7645ba2d99b06867675c6b3cd61fab9efc5540 WatchSource:0}: Error finding container 76792d9ca4104709c4f2e78c6c7645ba2d99b06867675c6b3cd61fab9efc5540: Status 404 returned error can't find the container with id 76792d9ca4104709c4f2e78c6c7645ba2d99b06867675c6b3cd61fab9efc5540 Nov 02 22:18:18.619351 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.619322 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" event={"ID":"3529b590-0d55-4064-87aa-7a02e242e454","Type":"ContainerDied","Data":"9c7f5cb8056ebd768a61732c0929fbe84b50795040ea81308e81a6ec7254b858"} Nov 02 22:18:18.619483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.619377 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c7f5cb8056ebd768a61732c0929fbe84b50795040ea81308e81a6ec7254b858" Nov 02 22:18:18.619483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.619455 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk" Nov 02 22:18:18.636086 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.636067 4309 scope.go:122] "RemoveContainer" containerID="34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd" Nov 02 22:18:18.636885 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:18:18.636856 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd\": container with ID starting with 34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd not found: ID does not exist" containerID="34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd" Nov 02 22:18:18.636980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.636896 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd"} err="failed to get container status \"34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd\": rpc error: code = NotFound desc = could not find container \"34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd\": container with ID starting with 34fc1f2d156f4d49cdd7103201ddfb106bb1841be01cb068b28abc77797526dd not found: ID does not exist" Nov 02 22:18:18.661758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.661664 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87"] Nov 02 22:18:18.667668 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:18.667645 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2fc3ce-openshift-default-7f9c65fbcc-dwv87"] Nov 02 22:18:19.633738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:19.633702 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" event={"ID":"a5748639-a17c-4ea1-9400-ac7fd3519cc7","Type":"ContainerStarted","Data":"dcfc309ca2fa804520d026dd426d75cd01e10f5e03a76b6d8f0e4b88008c6050"} Nov 02 22:18:19.633961 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:19.633771 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" event={"ID":"a5748639-a17c-4ea1-9400-ac7fd3519cc7","Type":"ContainerStarted","Data":"76792d9ca4104709c4f2e78c6c7645ba2d99b06867675c6b3cd61fab9efc5540"} Nov 02 22:18:19.684549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:19.684491 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" podStartSLOduration=2.684473623 podStartE2EDuration="2.684473623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:19.661338213 +0000 UTC m=+1227.599451431" watchObservedRunningTime="2026-11-02 22:18:19.684473623 +0000 UTC m=+1227.622586913" Nov 02 22:18:20.198379 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:20.198344 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1314c324-758d-4f87-985b-cab817bc6e31" path="/var/lib/kubelet/pods/1314c324-758d-4f87-985b-cab817bc6e31/volumes" Nov 02 22:18:20.268391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:20.268352 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:20.274372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:20.274346 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:20.645604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:20.645557 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:20.647251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:20.647210 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:18:21.683602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.683568 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt"] Nov 02 22:18:21.684444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.684424 4309 cpu_manager.go:414] "containerMap: removing container" podUID="3529b590-0d55-4064-87aa-7a02e242e454" containerName="revoke-keys" Nov 02 22:18:21.684444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.684444 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3529b590-0d55-4064-87aa-7a02e242e454" containerName="revoke-keys" Nov 02 22:18:21.684598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.684490 4309 cpu_manager.go:414] "containerMap: removing container" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" Nov 02 22:18:21.684598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.684495 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" Nov 02 22:18:21.684749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.684626 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="1314c324-758d-4f87-985b-cab817bc6e31" containerName="istio-proxy" Nov 02 22:18:21.684749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.684639 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="3529b590-0d55-4064-87aa-7a02e242e454" containerName="revoke-keys" Nov 02 22:18:21.690128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.690099 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:21.694013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.693987 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:18:21.695296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.695275 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:18:21.696023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.695987 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:18:21.696265 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.696246 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:18:21.698489 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.698468 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt"] Nov 02 22:18:21.739189 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.739147 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88q27\" (UniqueName: \"kubernetes.io/projected/3ac21b4e-8814-4d72-9b28-e2c33312e96d-kube-api-access-88q27\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-6pjwt\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:21.739358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.739273 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-6pjwt\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:21.739541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.739520 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-6pjwt\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:21.841594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.841545 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88q27\" (UniqueName: \"kubernetes.io/projected/3ac21b4e-8814-4d72-9b28-e2c33312e96d-kube-api-access-88q27\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-6pjwt\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:21.841782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.841641 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-6pjwt\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:21.841832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.841799 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-6pjwt\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:21.842419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.842394 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-6pjwt\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:21.843979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.843955 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-6pjwt\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:21.849538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:21.849511 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88q27\" (UniqueName: \"kubernetes.io/projected/3ac21b4e-8814-4d72-9b28-e2c33312e96d-kube-api-access-88q27\") pod \"maas-api-revoke-keys-e2e-mt-84a94060-6pjwt\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:22.011111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:22.011008 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:22.377615 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:18:22.377584 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac21b4e_8814_4d72_9b28_e2c33312e96d.slice/crio-a89847e5b1c7a56627a9649290b1b3df48afcadb534b5b7a1548a255ba584b0c WatchSource:0}: Error finding container a89847e5b1c7a56627a9649290b1b3df48afcadb534b5b7a1548a255ba584b0c: Status 404 returned error can't find the container with id a89847e5b1c7a56627a9649290b1b3df48afcadb534b5b7a1548a255ba584b0c Nov 02 22:18:22.379799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:22.379765 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt"] Nov 02 22:18:22.667810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:22.667726 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" event={"ID":"3ac21b4e-8814-4d72-9b28-e2c33312e96d","Type":"ContainerStarted","Data":"a89847e5b1c7a56627a9649290b1b3df48afcadb534b5b7a1548a255ba584b0c"} Nov 02 22:18:23.023712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:23.023599 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2"] Nov 02 22:18:23.029256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:23.029226 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb07e0a4-986b2"] Nov 02 22:18:23.059709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:23.059647 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7"] Nov 02 22:18:24.038346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:24.037413 4309 generic.go:356] "Generic (PLEG): container finished" podID="3ac21b4e-8814-4d72-9b28-e2c33312e96d" containerID="381f220b0bccb20bfc0242f1cfc6dec493469fcb808067785cf3681eb95a450d" exitCode=0 Nov 02 22:18:24.038346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:24.037497 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" event={"ID":"3ac21b4e-8814-4d72-9b28-e2c33312e96d","Type":"ContainerDied","Data":"381f220b0bccb20bfc0242f1cfc6dec493469fcb808067785cf3681eb95a450d"} Nov 02 22:18:24.204738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:24.204118 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ee6f97-4e20-4fc7-a7fb-bc1288836488" path="/var/lib/kubelet/pods/b1ee6f97-4e20-4fc7-a7fb-bc1288836488/volumes" Nov 02 22:18:25.236726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.236701 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:25.258562 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.258534 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac21b4e-8814-4d72-9b28-e2c33312e96d-kube-api-access-88q27\" (UniqueName: \"kubernetes.io/projected/3ac21b4e-8814-4d72-9b28-e2c33312e96d-kube-api-access-88q27\") pod \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " Nov 02 22:18:25.258728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.258645 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-openshift-ca\") pod \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " Nov 02 22:18:25.258789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.258749 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-xks-ca\") pod \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\" (UID: \"3ac21b4e-8814-4d72-9b28-e2c33312e96d\") " Nov 02 22:18:25.259015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.258993 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-openshift-ca" pod "3ac21b4e-8814-4d72-9b28-e2c33312e96d" (UID: "3ac21b4e-8814-4d72-9b28-e2c33312e96d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:25.259427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.259406 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:25.260928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.260903 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-xks-ca" pod "3ac21b4e-8814-4d72-9b28-e2c33312e96d" (UID: "3ac21b4e-8814-4d72-9b28-e2c33312e96d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:25.261044 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.260963 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac21b4e-8814-4d72-9b28-e2c33312e96d-kube-api-access-88q27" pod "3ac21b4e-8814-4d72-9b28-e2c33312e96d" (UID: "3ac21b4e-8814-4d72-9b28-e2c33312e96d"). InnerVolumeSpecName "kube-api-access-88q27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:25.368330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.368294 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88q27\" (UniqueName: \"kubernetes.io/projected/3ac21b4e-8814-4d72-9b28-e2c33312e96d-kube-api-access-88q27\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:25.368569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:25.368548 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ac21b4e-8814-4d72-9b28-e2c33312e96d-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:26.063656 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:26.063630 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" Nov 02 22:18:26.064561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:26.063622 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt" event={"ID":"3ac21b4e-8814-4d72-9b28-e2c33312e96d","Type":"ContainerDied","Data":"a89847e5b1c7a56627a9649290b1b3df48afcadb534b5b7a1548a255ba584b0c"} Nov 02 22:18:26.064722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:26.064626 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a89847e5b1c7a56627a9649290b1b3df48afcadb534b5b7a1548a255ba584b0c" Nov 02 22:18:27.022752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:27.022718 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2"] Nov 02 22:18:27.028592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:27.028562 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5cc3752-xt7z2"] Nov 02 22:18:28.031033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.030992 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc"] Nov 02 22:18:28.036180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.036152 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a28bbb0-q7btc"] Nov 02 22:18:28.060893 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.060853 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" podUID="ce66a98c-5f8d-4003-a769-9cdd31788f6e" containerName="payload-pre-processing" containerID="cri-o://60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4" gracePeriod=25 Nov 02 22:18:28.212209 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.210839 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a3209f-5dac-4ae0-a0d6-1de62f2212be" path="/var/lib/kubelet/pods/79a3209f-5dac-4ae0-a0d6-1de62f2212be/volumes" Nov 02 22:18:28.212209 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.211693 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0a2d65-7bc3-4760-90c6-0717a8a70502" path="/var/lib/kubelet/pods/df0a2d65-7bc3-4760-90c6-0717a8a70502/volumes" Nov 02 22:18:28.420596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.420571 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:18:28.571196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.571161 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce66a98c-5f8d-4003-a769-9cdd31788f6e-kube-api-access-b6pfg\" (UniqueName: \"kubernetes.io/projected/ce66a98c-5f8d-4003-a769-9cdd31788f6e-kube-api-access-b6pfg\") pod \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\" (UID: \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\") " Nov 02 22:18:28.571406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.571254 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce66a98c-5f8d-4003-a769-9cdd31788f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66a98c-5f8d-4003-a769-9cdd31788f6e-plugins-config-volume\") pod \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\" (UID: \"ce66a98c-5f8d-4003-a769-9cdd31788f6e\") " Nov 02 22:18:28.571628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.571593 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce66a98c-5f8d-4003-a769-9cdd31788f6e-plugins-config-volume" pod "ce66a98c-5f8d-4003-a769-9cdd31788f6e" (UID: "ce66a98c-5f8d-4003-a769-9cdd31788f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:28.572374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.572350 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce66a98c-5f8d-4003-a769-9cdd31788f6e-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:28.573594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.573571 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce66a98c-5f8d-4003-a769-9cdd31788f6e-kube-api-access-b6pfg" pod "ce66a98c-5f8d-4003-a769-9cdd31788f6e" (UID: "ce66a98c-5f8d-4003-a769-9cdd31788f6e"). InnerVolumeSpecName "kube-api-access-b6pfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:28.674371 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:28.674284 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6pfg\" (UniqueName: \"kubernetes.io/projected/ce66a98c-5f8d-4003-a769-9cdd31788f6e-kube-api-access-b6pfg\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:29.101009 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.100970 4309 generic.go:356] "Generic (PLEG): container finished" podID="ce66a98c-5f8d-4003-a769-9cdd31788f6e" containerID="60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4" exitCode=0 Nov 02 22:18:29.101455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.101023 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" Nov 02 22:18:29.101455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.101061 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" event={"ID":"ce66a98c-5f8d-4003-a769-9cdd31788f6e","Type":"ContainerDied","Data":"60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4"} Nov 02 22:18:29.101455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.101132 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7" event={"ID":"ce66a98c-5f8d-4003-a769-9cdd31788f6e","Type":"ContainerDied","Data":"c65590cd50e818df700d14b26b9fd116a0f7974e320d6bca4b3be77e145b4685"} Nov 02 22:18:29.101455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.101162 4309 scope.go:122] "RemoveContainer" containerID="60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4" Nov 02 22:18:29.114437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.114410 4309 scope.go:122] "RemoveContainer" containerID="60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4" Nov 02 22:18:29.114804 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:18:29.114776 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4\": container with ID starting with 60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4 not found: ID does not exist" containerID="60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4" Nov 02 22:18:29.114904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.114811 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4"} err="failed to get container status \"60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4\": rpc error: code = NotFound desc = could not find container \"60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4\": container with ID starting with 60f25615ac259fbfdad4c54e07b50c1ce1f42a8d01e904483327228d14a3e0f4 not found: ID does not exist" Nov 02 22:18:29.142236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.142195 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7"] Nov 02 22:18:29.145113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.145084 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0dd701-7dc5bbf94c-5g8k7"] Nov 02 22:18:29.500271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.500168 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw"] Nov 02 22:18:29.501732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.501691 4309 cpu_manager.go:414] "containerMap: removing container" podUID="ce66a98c-5f8d-4003-a769-9cdd31788f6e" containerName="payload-pre-processing" Nov 02 22:18:29.501732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.501733 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce66a98c-5f8d-4003-a769-9cdd31788f6e" containerName="payload-pre-processing" Nov 02 22:18:29.507885 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.507857 4309 cpu_manager.go:414] "containerMap: removing container" podUID="3ac21b4e-8814-4d72-9b28-e2c33312e96d" containerName="revoke-keys" Nov 02 22:18:29.507885 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.507883 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac21b4e-8814-4d72-9b28-e2c33312e96d" containerName="revoke-keys" Nov 02 22:18:29.508202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.508188 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac21b4e-8814-4d72-9b28-e2c33312e96d" containerName="revoke-keys" Nov 02 22:18:29.508295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.508214 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce66a98c-5f8d-4003-a769-9cdd31788f6e" containerName="payload-pre-processing" Nov 02 22:18:29.526074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.514661 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw"] Nov 02 22:18:29.526074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.514796 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.526074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.520069 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:18:29.526074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.520148 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:18:29.526074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.520617 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:18:29.526074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.520749 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:18:29.593896 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.593845 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.594186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.594157 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.594424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.594403 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrl4\" (UniqueName: \"kubernetes.io/projected/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-kube-api-access-rjrl4\") pod \"maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.696413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.696372 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.696608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.696540 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.696740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.696720 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrl4\" (UniqueName: \"kubernetes.io/projected/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-kube-api-access-rjrl4\") pod \"maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.697058 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.697038 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.699043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.699020 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.703995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.703975 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrl4\" (UniqueName: \"kubernetes.io/projected/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-kube-api-access-rjrl4\") pod \"maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:29.837174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:29.837138 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:30.135381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:30.135349 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw"] Nov 02 22:18:30.135772 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:18:30.135749 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e6945f_69d8_4d6d_9401_6e3598a2d3d1.slice/crio-2ed4926fb20b271e339bb29d7525919dff324fbd10dae72d77c5978d7714529f WatchSource:0}: Error finding container 2ed4926fb20b271e339bb29d7525919dff324fbd10dae72d77c5978d7714529f: Status 404 returned error can't find the container with id 2ed4926fb20b271e339bb29d7525919dff324fbd10dae72d77c5978d7714529f Nov 02 22:18:30.193764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:30.193730 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce66a98c-5f8d-4003-a769-9cdd31788f6e" path="/var/lib/kubelet/pods/ce66a98c-5f8d-4003-a769-9cdd31788f6e/volumes" Nov 02 22:18:31.126242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:31.126201 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" event={"ID":"17e6945f-69d8-4d6d-9401-6e3598a2d3d1","Type":"ContainerStarted","Data":"b9f8dd0e5aae76f085d10e45148dad86ccbe6268f4bc08479b614232c4bac390"} Nov 02 22:18:31.126473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:31.126268 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" event={"ID":"17e6945f-69d8-4d6d-9401-6e3598a2d3d1","Type":"ContainerStarted","Data":"2ed4926fb20b271e339bb29d7525919dff324fbd10dae72d77c5978d7714529f"} Nov 02 22:18:31.142307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:31.142262 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" podStartSLOduration=2.142249268 podStartE2EDuration="2.142249268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:31.141298966 +0000 UTC m=+1239.079412186" watchObservedRunningTime="2026-11-02 22:18:31.142249268 +0000 UTC m=+1239.080362485" Nov 02 22:18:32.030756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:32.030719 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9"] Nov 02 22:18:32.036834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:32.036803 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac617154-9mcj9"] Nov 02 22:18:32.139168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:32.139136 4309 generic.go:356] "Generic (PLEG): container finished" podID="17e6945f-69d8-4d6d-9401-6e3598a2d3d1" containerID="b9f8dd0e5aae76f085d10e45148dad86ccbe6268f4bc08479b614232c4bac390" exitCode=0 Nov 02 22:18:32.139374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:32.139236 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" event={"ID":"17e6945f-69d8-4d6d-9401-6e3598a2d3d1","Type":"ContainerDied","Data":"b9f8dd0e5aae76f085d10e45148dad86ccbe6268f4bc08479b614232c4bac390"} Nov 02 22:18:32.196138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:32.196091 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901b6eb1-9e1b-4442-bdc5-e5cda2707940" path="/var/lib/kubelet/pods/901b6eb1-9e1b-4442-bdc5-e5cda2707940/volumes" Nov 02 22:18:33.032013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:33.031959 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm"] Nov 02 22:18:33.062192 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:33.062101 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f934efa9-7glmm"] Nov 02 22:18:33.890317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:33.890234 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt"] Nov 02 22:18:33.908140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:33.908092 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84a94060-6pjwt"] Nov 02 22:18:33.940206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:33.940175 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:34.115499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.114424 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-xks-ca\") pod \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " Nov 02 22:18:34.115499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.114737 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-kube-api-access-rjrl4\" (UniqueName: \"kubernetes.io/projected/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-kube-api-access-rjrl4\") pod \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " Nov 02 22:18:34.115499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.114794 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-openshift-ca\") pod \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\" (UID: \"17e6945f-69d8-4d6d-9401-6e3598a2d3d1\") " Nov 02 22:18:34.116607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.116580 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-openshift-ca" pod "17e6945f-69d8-4d6d-9401-6e3598a2d3d1" (UID: "17e6945f-69d8-4d6d-9401-6e3598a2d3d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:34.118605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.118582 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-kube-api-access-rjrl4" pod "17e6945f-69d8-4d6d-9401-6e3598a2d3d1" (UID: "17e6945f-69d8-4d6d-9401-6e3598a2d3d1"). InnerVolumeSpecName "kube-api-access-rjrl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:34.126126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.126097 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-xks-ca" pod "17e6945f-69d8-4d6d-9401-6e3598a2d3d1" (UID: "17e6945f-69d8-4d6d-9401-6e3598a2d3d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:18:34.212042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.211960 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac21b4e-8814-4d72-9b28-e2c33312e96d" path="/var/lib/kubelet/pods/3ac21b4e-8814-4d72-9b28-e2c33312e96d/volumes" Nov 02 22:18:34.212920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.212900 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58553e2b-8414-4b6f-8682-0547655fa062" path="/var/lib/kubelet/pods/58553e2b-8414-4b6f-8682-0547655fa062/volumes" Nov 02 22:18:34.216510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.216484 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjrl4\" (UniqueName: \"kubernetes.io/projected/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-kube-api-access-rjrl4\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:34.216510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.216512 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:34.216650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.216522 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e6945f-69d8-4d6d-9401-6e3598a2d3d1-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:34.284078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.284036 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" event={"ID":"17e6945f-69d8-4d6d-9401-6e3598a2d3d1","Type":"ContainerDied","Data":"2ed4926fb20b271e339bb29d7525919dff324fbd10dae72d77c5978d7714529f"} Nov 02 22:18:34.284259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.284106 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ed4926fb20b271e339bb29d7525919dff324fbd10dae72d77c5978d7714529f" Nov 02 22:18:34.284259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:34.284176 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw" Nov 02 22:18:36.938228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:36.938194 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk"] Nov 02 22:18:36.943200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:36.943172 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0dd701-btmqk"] Nov 02 22:18:37.044784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:37.044717 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz"] Nov 02 22:18:37.051201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:37.051168 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f6704a-qrqrz"] Nov 02 22:18:38.518695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:38.513643 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3529b590-0d55-4064-87aa-7a02e242e454" path="/var/lib/kubelet/pods/3529b590-0d55-4064-87aa-7a02e242e454/volumes" Nov 02 22:18:38.518695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:38.514417 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e154eb-fb69-4a0a-b311-f4db51e64c21" path="/var/lib/kubelet/pods/f4e154eb-fb69-4a0a-b311-f4db51e64c21/volumes" Nov 02 22:18:39.037799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:39.037759 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv"] Nov 02 22:18:39.047038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:39.047001 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22358063-f26pv"] Nov 02 22:18:39.205827 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:39.205781 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:39.556695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:39.555784 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:39.619835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:39.619793 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:40.071703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.070841 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:40.245697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.244726 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": context deadline exceeded" Nov 02 22:18:40.528449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.528342 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 22:18:40.601300 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.536647 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 22:18:40.601300 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.556013 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:40.602079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.583257 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4fd685b-1edf-42fb-b508-bfb91a72d32c" path="/var/lib/kubelet/pods/e4fd685b-1edf-42fb-b508-bfb91a72d32c/volumes" Nov 02 22:18:40.602885 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.602862 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:18:40.620869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.620816 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 22:18:40.633178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.633138 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:18:40.840542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:40.840495 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:41.074382 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:41.074334 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Nov 02 22:18:41.075596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:41.075572 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:18:41.168025 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:41.167895 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:41.283999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:41.283939 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:18:41.306231 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:41.305981 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:18:41.405404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:41.405371 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:18:41.405706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:41.405660 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:18:41.405901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:41.405887 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:18:41.478244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:41.478166 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:18:41.647940 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:18:41.647869 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:18:43.029312 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:43.029274 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk"] Nov 02 22:18:43.034230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:43.034202 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce66a83-qhrfk"] Nov 02 22:18:44.207695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:44.206865 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405643ba-bf63-46f9-b0c4-d57622186f6a" path="/var/lib/kubelet/pods/405643ba-bf63-46f9-b0c4-d57622186f6a/volumes" Nov 02 22:18:44.550847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:44.550751 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4"] Nov 02 22:18:44.551035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:44.551005 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" containerID="cri-o://c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925" gracePeriod=30 Nov 02 22:18:46.330423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:46.330386 4309 patch_prober.go:29] interesting pod/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:18:46.330848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:46.330462 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:18:49.955724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:49.951361 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:18:50.027429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.027392 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-envoy\") pod \"c0762470-8657-4a0e-b29f-e6064a65d2ca\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " Nov 02 22:18:50.027605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.027573 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0762470-8657-4a0e-b29f-e6064a65d2ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0762470-8657-4a0e-b29f-e6064a65d2ca-istiod-ca-cert\") pod \"c0762470-8657-4a0e-b29f-e6064a65d2ca\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " Nov 02 22:18:50.027771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.027627 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-credential-socket\") pod \"c0762470-8657-4a0e-b29f-e6064a65d2ca\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " Nov 02 22:18:50.027771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.027666 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-socket\") pod \"c0762470-8657-4a0e-b29f-e6064a65d2ca\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " Nov 02 22:18:50.027890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.027772 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-data\") pod \"c0762470-8657-4a0e-b29f-e6064a65d2ca\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " Nov 02 22:18:50.027890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.027802 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-token\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-token\") pod \"c0762470-8657-4a0e-b29f-e6064a65d2ca\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " Nov 02 22:18:50.027890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.027852 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-podinfo\") pod \"c0762470-8657-4a0e-b29f-e6064a65d2ca\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " Nov 02 22:18:50.028031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.027890 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-kube-api-access-4b6fc\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-kube-api-access-4b6fc\") pod \"c0762470-8657-4a0e-b29f-e6064a65d2ca\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " Nov 02 22:18:50.028031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.027961 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-credential-socket" pod "c0762470-8657-4a0e-b29f-e6064a65d2ca" (UID: "c0762470-8657-4a0e-b29f-e6064a65d2ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:50.028134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.028033 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-socket" pod "c0762470-8657-4a0e-b29f-e6064a65d2ca" (UID: "c0762470-8657-4a0e-b29f-e6064a65d2ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:50.028134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.028045 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-certs\") pod \"c0762470-8657-4a0e-b29f-e6064a65d2ca\" (UID: \"c0762470-8657-4a0e-b29f-e6064a65d2ca\") " Nov 02 22:18:50.028134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.028055 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0762470-8657-4a0e-b29f-e6064a65d2ca-istiod-ca-cert" pod "c0762470-8657-4a0e-b29f-e6064a65d2ca" (UID: "c0762470-8657-4a0e-b29f-e6064a65d2ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:18:50.028355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.028329 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-data" pod "c0762470-8657-4a0e-b29f-e6064a65d2ca" (UID: "c0762470-8657-4a0e-b29f-e6064a65d2ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:50.029281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.028595 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-certs" pod "c0762470-8657-4a0e-b29f-e6064a65d2ca" (UID: "c0762470-8657-4a0e-b29f-e6064a65d2ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:50.029281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.029170 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0762470-8657-4a0e-b29f-e6064a65d2ca-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:50.029281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.029193 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:50.029281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.029208 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:50.029281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.029222 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:50.029281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.029237 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:50.031451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.031425 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-envoy" pod "c0762470-8657-4a0e-b29f-e6064a65d2ca" (UID: "c0762470-8657-4a0e-b29f-e6064a65d2ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:18:50.031451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.031433 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-podinfo" pod "c0762470-8657-4a0e-b29f-e6064a65d2ca" (UID: "c0762470-8657-4a0e-b29f-e6064a65d2ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:18:50.031857 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.031821 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-kube-api-access-4b6fc" pod "c0762470-8657-4a0e-b29f-e6064a65d2ca" (UID: "c0762470-8657-4a0e-b29f-e6064a65d2ca"). InnerVolumeSpecName "kube-api-access-4b6fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:50.031968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.031944 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-token" pod "c0762470-8657-4a0e-b29f-e6064a65d2ca" (UID: "c0762470-8657-4a0e-b29f-e6064a65d2ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:18:50.131314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.131279 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:50.131314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.131310 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:50.131512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.131325 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b6fc\" (UniqueName: \"kubernetes.io/projected/c0762470-8657-4a0e-b29f-e6064a65d2ca-kube-api-access-4b6fc\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:50.131512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.131338 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0762470-8657-4a0e-b29f-e6064a65d2ca-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:18:50.557034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.556935 4309 generic.go:356] "Generic (PLEG): container finished" podID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerID="c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925" exitCode=0 Nov 02 22:18:50.557034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.557020 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" Nov 02 22:18:50.557270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.557026 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" event={"ID":"c0762470-8657-4a0e-b29f-e6064a65d2ca","Type":"ContainerDied","Data":"c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925"} Nov 02 22:18:50.557270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.557094 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4" event={"ID":"c0762470-8657-4a0e-b29f-e6064a65d2ca","Type":"ContainerDied","Data":"3c72521e8033bf4d9d2f6b955a132c5b29b31333cbbf63d5e304846d25f9eb28"} Nov 02 22:18:50.557270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.557124 4309 scope.go:122] "RemoveContainer" containerID="c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925" Nov 02 22:18:50.571884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.571861 4309 scope.go:122] "RemoveContainer" containerID="213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29" Nov 02 22:18:50.584711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.582290 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4"] Nov 02 22:18:50.593929 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.593898 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0dd701-openshift-default-794459ddff-v4rm4"] Nov 02 22:18:50.604910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.603696 4309 scope.go:122] "RemoveContainer" containerID="c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925" Nov 02 22:18:50.604910 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:18:50.604208 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925\": container with ID starting with c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925 not found: ID does not exist" containerID="c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925" Nov 02 22:18:50.604910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.604287 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925"} err="failed to get container status \"c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925\": rpc error: code = NotFound desc = could not find container \"c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925\": container with ID starting with c290adfe8bb363ee0412da500f26fba2dc165b41d1c4c8cc0977833dbf055925 not found: ID does not exist" Nov 02 22:18:50.604910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.604315 4309 scope.go:122] "RemoveContainer" containerID="213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29" Nov 02 22:18:50.604910 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:18:50.604640 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29\": container with ID starting with 213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29 not found: ID does not exist" containerID="213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29" Nov 02 22:18:50.605227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:50.604669 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29"} err="failed to get container status \"213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29\": rpc error: code = NotFound desc = could not find container \"213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29\": container with ID starting with 213a8e20cc95a403390a9d8d367b2008995cd5a3c794772c7fd96cd6d2a88d29 not found: ID does not exist" Nov 02 22:18:51.049698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:51.046932 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc"] Nov 02 22:18:51.057709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:51.057407 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp"] Nov 02 22:18:51.063920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:51.063889 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f052da-cqblc"] Nov 02 22:18:51.069099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:51.069070 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0f9863c-9kglp"] Nov 02 22:18:52.213040 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.212964 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43039834-e511-4d83-b65d-51e6233b1b7a" path="/var/lib/kubelet/pods/43039834-e511-4d83-b65d-51e6233b1b7a/volumes" Nov 02 22:18:52.213899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.213876 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa204759-fc38-41ee-bc04-f56e56d5de79" path="/var/lib/kubelet/pods/aa204759-fc38-41ee-bc04-f56e56d5de79/volumes" Nov 02 22:18:52.214735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.214712 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" path="/var/lib/kubelet/pods/c0762470-8657-4a0e-b29f-e6064a65d2ca/volumes" Nov 02 22:18:52.403853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.403813 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd"] Nov 02 22:18:52.405135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.405109 4309 cpu_manager.go:414] "containerMap: removing container" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" Nov 02 22:18:52.405135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.405133 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" Nov 02 22:18:52.405341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.405168 4309 cpu_manager.go:414] "containerMap: removing container" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" Nov 02 22:18:52.405341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.405176 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" Nov 02 22:18:52.405341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.405236 4309 cpu_manager.go:414] "containerMap: removing container" podUID="17e6945f-69d8-4d6d-9401-6e3598a2d3d1" containerName="revoke-keys" Nov 02 22:18:52.405341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.405247 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e6945f-69d8-4d6d-9401-6e3598a2d3d1" containerName="revoke-keys" Nov 02 22:18:52.405624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.405452 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" Nov 02 22:18:52.405624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.405474 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e6945f-69d8-4d6d-9401-6e3598a2d3d1" containerName="revoke-keys" Nov 02 22:18:52.413492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.413208 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:18:52.414834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.414807 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd"] Nov 02 22:18:52.416955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.416936 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e30968\"" Nov 02 22:18:52.418040 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.418011 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e30968-dockercfg-2wds5\"" Nov 02 22:18:52.593708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.592516 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e97fdbf-c24b-4727-b895-18915dc6b1a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd\" (UID: \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:18:52.655709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.653461 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzq6\" (UniqueName: \"kubernetes.io/projected/6e97fdbf-c24b-4727-b895-18915dc6b1a0-kube-api-access-nhzq6\") pod \"payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd\" (UID: \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:18:52.757698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.757154 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e97fdbf-c24b-4727-b895-18915dc6b1a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd\" (UID: \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:18:52.757698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.757424 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzq6\" (UniqueName: \"kubernetes.io/projected/6e97fdbf-c24b-4727-b895-18915dc6b1a0-kube-api-access-nhzq6\") pod \"payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd\" (UID: \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:18:52.774463 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.774201 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e97fdbf-c24b-4727-b895-18915dc6b1a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd\" (UID: \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:18:52.806740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:52.805843 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzq6\" (UniqueName: \"kubernetes.io/projected/6e97fdbf-c24b-4727-b895-18915dc6b1a0-kube-api-access-nhzq6\") pod \"payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd\" (UID: \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:18:53.035724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:53.035599 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:18:53.432699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:53.430206 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd"] Nov 02 22:18:53.856591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:53.856548 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" event={"ID":"6e97fdbf-c24b-4727-b895-18915dc6b1a0","Type":"ContainerStarted","Data":"a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a"} Nov 02 22:18:53.856876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:53.856615 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" event={"ID":"6e97fdbf-c24b-4727-b895-18915dc6b1a0","Type":"ContainerStarted","Data":"da0eb3b6457d2d54304d7788c03f61b895284cad3b07cb1ab0b1ef710ccb25af"} Nov 02 22:18:53.856876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:53.856645 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:18:53.877907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:53.877815 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" podStartSLOduration=1.877800266 podStartE2EDuration="1.877800266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:18:53.872216664 +0000 UTC m=+1261.810329883" watchObservedRunningTime="2026-11-02 22:18:53.877800266 +0000 UTC m=+1261.815913484" Nov 02 22:18:55.342000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.341964 4309 scope.go:122] "RemoveContainer" containerID="ca21569b1aa0ec532bb48b050e81f601c3f1b1b929b038bb7d71f2d877f69f02" Nov 02 22:18:55.357333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.357305 4309 scope.go:122] "RemoveContainer" containerID="8a86b3a436eca343f1787d15efafb45343fcd6cff6417b1bf0ea9b48637a4be0" Nov 02 22:18:55.378113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.378088 4309 scope.go:122] "RemoveContainer" containerID="e6a8c50fbdce5fa2180b06d6306ab208928ec41c93fcdb3e258a01e3d61be9cd" Nov 02 22:18:55.390132 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.390109 4309 scope.go:122] "RemoveContainer" containerID="e7b42638c8f2075d19738ccfabc09f8c1e7b010fc1b4ba5568f0c0b592a442f5" Nov 02 22:18:55.402821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.402795 4309 scope.go:122] "RemoveContainer" containerID="f3d510132c319077c736c140fc114bb3bf2d8ee9f322fc326e6c0277fb61ea38" Nov 02 22:18:55.416221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.416196 4309 scope.go:122] "RemoveContainer" containerID="eb790522427fb2f4f6b4180aa4645a5a443377ce9ce3aaf57278610e339c19c8" Nov 02 22:18:55.428255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.428230 4309 scope.go:122] "RemoveContainer" containerID="0e4fd11ec9610343659ebfc57249168f02a2e87ecbbf9dbc557a3d7b42dfb5c0" Nov 02 22:18:55.439255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.439232 4309 scope.go:122] "RemoveContainer" containerID="f7b9d935f86072ead646588cb60506b6de10af0f59d9e4f734b3a9783c479a51" Nov 02 22:18:55.450457 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.450435 4309 scope.go:122] "RemoveContainer" containerID="accec6faaf86f93df9755517facd0f8d38d634ceaa18532c310cd237d67b40b2" Nov 02 22:18:55.468237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:55.468211 4309 scope.go:122] "RemoveContainer" containerID="0cf2cda99907725215d91ba8374c7f72e74357d014fb63cb0e550c777d99ae86" Nov 02 22:18:58.984921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:58.972811 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw"] Nov 02 22:18:58.985409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:18:58.984949 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d9416f-mhjmw"] Nov 02 22:19:00.192020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:00.191989 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e6945f-69d8-4d6d-9401-6e3598a2d3d1" path="/var/lib/kubelet/pods/17e6945f-69d8-4d6d-9401-6e3598a2d3d1/volumes" Nov 02 22:19:01.027992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:01.027953 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf"] Nov 02 22:19:01.031870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:01.031839 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56138600-8gwbf"] Nov 02 22:19:02.194318 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:02.194283 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc68026-e119-4d93-addc-9bb44e9dbe9f" path="/var/lib/kubelet/pods/1dc68026-e119-4d93-addc-9bb44e9dbe9f/volumes" Nov 02 22:19:04.872160 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:04.872129 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:19:07.223119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.223084 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq"] Nov 02 22:19:07.224597 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.224573 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0762470-8657-4a0e-b29f-e6064a65d2ca" containerName="istio-proxy" Nov 02 22:19:07.269462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.265827 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.283702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.276470 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:19:07.283702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.278532 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:19:07.283702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.280014 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:19:07.283702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.280353 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:19:07.295703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.295188 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.295703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.295310 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7p4h\" (UniqueName: \"kubernetes.io/projected/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-kube-api-access-k7p4h\") pod \"maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.295703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.295389 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.327563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.327537 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq"] Nov 02 22:19:07.396993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.396962 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.397202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.397051 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7p4h\" (UniqueName: \"kubernetes.io/projected/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-kube-api-access-k7p4h\") pod \"maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.397202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.397119 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.398332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.398311 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.401753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.401733 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.406873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.406848 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7p4h\" (UniqueName: \"kubernetes.io/projected/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-kube-api-access-k7p4h\") pod \"maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.628270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.628237 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:07.915622 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:19:07.915590 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42cf933_c88e_4a4a_a3f7_74f669f71f8d.slice/crio-ad5bd5051a7f303c536f2380afaf23cdce4eb300725ac87793abd821f913db73 WatchSource:0}: Error finding container ad5bd5051a7f303c536f2380afaf23cdce4eb300725ac87793abd821f913db73: Status 404 returned error can't find the container with id ad5bd5051a7f303c536f2380afaf23cdce4eb300725ac87793abd821f913db73 Nov 02 22:19:07.916195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:07.916166 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq"] Nov 02 22:19:08.057110 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:08.057076 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" event={"ID":"b42cf933-c88e-4a4a-a3f7-74f669f71f8d","Type":"ContainerStarted","Data":"ad5bd5051a7f303c536f2380afaf23cdce4eb300725ac87793abd821f913db73"} Nov 02 22:19:09.068689 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:09.068641 4309 generic.go:356] "Generic (PLEG): container finished" podID="b42cf933-c88e-4a4a-a3f7-74f669f71f8d" containerID="095388bc3637d785d51b111bdef55fc9dd1d8877cbe4ee3b0d611d5338990e11" exitCode=0 Nov 02 22:19:09.069176 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:09.068766 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" event={"ID":"b42cf933-c88e-4a4a-a3f7-74f669f71f8d","Type":"ContainerDied","Data":"095388bc3637d785d51b111bdef55fc9dd1d8877cbe4ee3b0d611d5338990e11"} Nov 02 22:19:10.292081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.292050 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:10.362250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.362148 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-openshift-ca\") pod \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " Nov 02 22:19:10.362250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.362220 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-xks-ca\") pod \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " Nov 02 22:19:10.362610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.362314 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-kube-api-access-k7p4h\" (UniqueName: \"kubernetes.io/projected/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-kube-api-access-k7p4h\") pod \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\" (UID: \"b42cf933-c88e-4a4a-a3f7-74f669f71f8d\") " Nov 02 22:19:10.362893 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.362839 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-openshift-ca" pod "b42cf933-c88e-4a4a-a3f7-74f669f71f8d" (UID: "b42cf933-c88e-4a4a-a3f7-74f669f71f8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:10.365758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.365732 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:10.370187 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.370157 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-kube-api-access-k7p4h" pod "b42cf933-c88e-4a4a-a3f7-74f669f71f8d" (UID: "b42cf933-c88e-4a4a-a3f7-74f669f71f8d"). InnerVolumeSpecName "kube-api-access-k7p4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:10.372826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.372799 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-xks-ca" pod "b42cf933-c88e-4a4a-a3f7-74f669f71f8d" (UID: "b42cf933-c88e-4a4a-a3f7-74f669f71f8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:10.467506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.467473 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:10.467506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:10.467501 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7p4h\" (UniqueName: \"kubernetes.io/projected/b42cf933-c88e-4a4a-a3f7-74f669f71f8d-kube-api-access-k7p4h\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:11.092461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:11.092432 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" Nov 02 22:19:11.092461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:11.092448 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq" event={"ID":"b42cf933-c88e-4a4a-a3f7-74f669f71f8d","Type":"ContainerDied","Data":"ad5bd5051a7f303c536f2380afaf23cdce4eb300725ac87793abd821f913db73"} Nov 02 22:19:11.092714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:11.092494 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5bd5051a7f303c536f2380afaf23cdce4eb300725ac87793abd821f913db73" Nov 02 22:19:11.602852 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:19:11.602808 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:19:12.024059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:12.023962 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk"] Nov 02 22:19:12.028980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:12.028950 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-965669d9-zxqrk"] Nov 02 22:19:12.193982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:12.193949 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221fb35c-8484-4a36-93a1-4ad9904ceee8" path="/var/lib/kubelet/pods/221fb35c-8484-4a36-93a1-4ad9904ceee8/volumes" Nov 02 22:19:17.537696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:17.535305 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g"] Nov 02 22:19:22.377046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.377000 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz"] Nov 02 22:19:22.378001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.377978 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b42cf933-c88e-4a4a-a3f7-74f669f71f8d" containerName="revoke-keys" Nov 02 22:19:22.378001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.377999 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42cf933-c88e-4a4a-a3f7-74f669f71f8d" containerName="revoke-keys" Nov 02 22:19:22.378166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.378127 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42cf933-c88e-4a4a-a3f7-74f669f71f8d" containerName="revoke-keys" Nov 02 22:19:22.409726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.401859 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz"] Nov 02 22:19:22.409726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.401982 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.410038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.409998 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:19:22.416698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.410871 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:19:22.416698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.410987 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:19:22.416698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.411133 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:19:22.493696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.493081 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfn2v\" (UniqueName: \"kubernetes.io/projected/356ea8f5-7786-44c5-90c3-064b8dd2c626-kube-api-access-xfn2v\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.493696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.493202 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.493696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.493381 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.536194 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.536157 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" podUID="8b051708-5a84-4f0f-ba1a-b6ea2ff48099" containerName="payload-pre-processing" containerID="cri-o://89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d" gracePeriod=25 Nov 02 22:19:22.597700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.594884 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfn2v\" (UniqueName: \"kubernetes.io/projected/356ea8f5-7786-44c5-90c3-064b8dd2c626-kube-api-access-xfn2v\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.597700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.595004 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.597700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.595188 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.597700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.596240 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.669107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.669031 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfn2v\" (UniqueName: \"kubernetes.io/projected/356ea8f5-7786-44c5-90c3-064b8dd2c626-kube-api-access-xfn2v\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.669722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.669641 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:22.744213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:22.744173 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:23.018222 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.018053 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:19:23.134002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.133972 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-plugins-config-volume\") pod \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\" (UID: \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\") " Nov 02 22:19:23.134283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.134261 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-kube-api-access-jtqmm\" (UniqueName: \"kubernetes.io/projected/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-kube-api-access-jtqmm\") pod \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\" (UID: \"8b051708-5a84-4f0f-ba1a-b6ea2ff48099\") " Nov 02 22:19:23.134417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.134395 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-plugins-config-volume" pod "8b051708-5a84-4f0f-ba1a-b6ea2ff48099" (UID: "8b051708-5a84-4f0f-ba1a-b6ea2ff48099"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:23.135061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.135038 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:23.136441 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.136415 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-kube-api-access-jtqmm" pod "8b051708-5a84-4f0f-ba1a-b6ea2ff48099" (UID: "8b051708-5a84-4f0f-ba1a-b6ea2ff48099"). InnerVolumeSpecName "kube-api-access-jtqmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:23.237423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.237388 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtqmm\" (UniqueName: \"kubernetes.io/projected/8b051708-5a84-4f0f-ba1a-b6ea2ff48099-kube-api-access-jtqmm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:23.292427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.292389 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz"] Nov 02 22:19:23.295643 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:19:23.295608 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356ea8f5_7786_44c5_90c3_064b8dd2c626.slice/crio-a8629d10ffa6bca7ca217ca662dea9897695253211ef16b1e78d12916947f311 WatchSource:0}: Error finding container a8629d10ffa6bca7ca217ca662dea9897695253211ef16b1e78d12916947f311: Status 404 returned error can't find the container with id a8629d10ffa6bca7ca217ca662dea9897695253211ef16b1e78d12916947f311 Nov 02 22:19:23.568124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.555802 4309 generic.go:356] "Generic (PLEG): container finished" podID="8b051708-5a84-4f0f-ba1a-b6ea2ff48099" containerID="89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d" exitCode=0 Nov 02 22:19:23.568124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.556002 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" event={"ID":"8b051708-5a84-4f0f-ba1a-b6ea2ff48099","Type":"ContainerDied","Data":"89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d"} Nov 02 22:19:23.568124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.556061 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" event={"ID":"8b051708-5a84-4f0f-ba1a-b6ea2ff48099","Type":"ContainerDied","Data":"a7a2935694e49793a4381ff6c1986d4b83870862591979c22ae9b70050525e9c"} Nov 02 22:19:23.568124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.556112 4309 scope.go:122] "RemoveContainer" containerID="89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d" Nov 02 22:19:23.568124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.562005 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g" Nov 02 22:19:23.625697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.625067 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" event={"ID":"356ea8f5-7786-44c5-90c3-064b8dd2c626","Type":"ContainerStarted","Data":"a8629d10ffa6bca7ca217ca662dea9897695253211ef16b1e78d12916947f311"} Nov 02 22:19:23.727695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.722903 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g"] Nov 02 22:19:23.727695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.725816 4309 scope.go:122] "RemoveContainer" containerID="89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d" Nov 02 22:19:23.727695 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:19:23.726304 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d\": container with ID starting with 89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d not found: ID does not exist" containerID="89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d" Nov 02 22:19:23.727695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.726335 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d"} err="failed to get container status \"89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d\": rpc error: code = NotFound desc = could not find container \"89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d\": container with ID starting with 89179a6357c149ee4f28343fdb7be2a4189a212c48954f8170f0a41dcf682b0d not found: ID does not exist" Nov 02 22:19:23.740208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:23.740174 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14dae2-b466f75c5-kqb7g"] Nov 02 22:19:24.193881 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:24.193848 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b051708-5a84-4f0f-ba1a-b6ea2ff48099" path="/var/lib/kubelet/pods/8b051708-5a84-4f0f-ba1a-b6ea2ff48099/volumes" Nov 02 22:19:24.640855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:24.640822 4309 generic.go:356] "Generic (PLEG): container finished" podID="356ea8f5-7786-44c5-90c3-064b8dd2c626" containerID="d042a70ca66d57ac619f5a9a56fc03fb1f5f6b198e4e249578c0f913ee0f5324" exitCode=0 Nov 02 22:19:24.641285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:24.640906 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" event={"ID":"356ea8f5-7786-44c5-90c3-064b8dd2c626","Type":"ContainerDied","Data":"d042a70ca66d57ac619f5a9a56fc03fb1f5f6b198e4e249578c0f913ee0f5324"} Nov 02 22:19:25.431699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:25.424442 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj"] Nov 02 22:19:25.431699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:25.424706 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" containerID="cri-o://354ca6fccc191fa28de211107c89c89ec522b9386ce90fdf8f5aa6fd55896571" gracePeriod=30 Nov 02 22:19:25.912103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:25.912068 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:26.010306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.008772 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-xks-ca\") pod \"356ea8f5-7786-44c5-90c3-064b8dd2c626\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " Nov 02 22:19:26.010306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.009063 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356ea8f5-7786-44c5-90c3-064b8dd2c626-kube-api-access-xfn2v\" (UniqueName: \"kubernetes.io/projected/356ea8f5-7786-44c5-90c3-064b8dd2c626-kube-api-access-xfn2v\") pod \"356ea8f5-7786-44c5-90c3-064b8dd2c626\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " Nov 02 22:19:26.010306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.009231 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-openshift-ca\") pod \"356ea8f5-7786-44c5-90c3-064b8dd2c626\" (UID: \"356ea8f5-7786-44c5-90c3-064b8dd2c626\") " Nov 02 22:19:26.011328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.011285 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-openshift-ca" pod "356ea8f5-7786-44c5-90c3-064b8dd2c626" (UID: "356ea8f5-7786-44c5-90c3-064b8dd2c626"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:26.014628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.014596 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-xks-ca" pod "356ea8f5-7786-44c5-90c3-064b8dd2c626" (UID: "356ea8f5-7786-44c5-90c3-064b8dd2c626"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:26.017471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.017370 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356ea8f5-7786-44c5-90c3-064b8dd2c626-kube-api-access-xfn2v" pod "356ea8f5-7786-44c5-90c3-064b8dd2c626" (UID: "356ea8f5-7786-44c5-90c3-064b8dd2c626"). InnerVolumeSpecName "kube-api-access-xfn2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:26.112114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.112072 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:26.112114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.112114 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfn2v\" (UniqueName: \"kubernetes.io/projected/356ea8f5-7786-44c5-90c3-064b8dd2c626-kube-api-access-xfn2v\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:26.112366 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.112134 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/356ea8f5-7786-44c5-90c3-064b8dd2c626-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:26.552754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.552719 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j"] Nov 02 22:19:26.553759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.553737 4309 cpu_manager.go:414] "containerMap: removing container" podUID="356ea8f5-7786-44c5-90c3-064b8dd2c626" containerName="revoke-keys" Nov 02 22:19:26.553840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.553761 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356ea8f5-7786-44c5-90c3-064b8dd2c626" containerName="revoke-keys" Nov 02 22:19:26.553840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.553790 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8b051708-5a84-4f0f-ba1a-b6ea2ff48099" containerName="payload-pre-processing" Nov 02 22:19:26.553840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.553798 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b051708-5a84-4f0f-ba1a-b6ea2ff48099" containerName="payload-pre-processing" Nov 02 22:19:26.554041 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.554028 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="356ea8f5-7786-44c5-90c3-064b8dd2c626" containerName="revoke-keys" Nov 02 22:19:26.554077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.554048 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b051708-5a84-4f0f-ba1a-b6ea2ff48099" containerName="payload-pre-processing" Nov 02 22:19:26.561062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.561033 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.563774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.563750 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j"] Nov 02 22:19:26.636000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.635961 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f652337-8lc9j\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.636230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.636204 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqcn\" (UniqueName: \"kubernetes.io/projected/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-kube-api-access-zkqcn\") pod \"maas-api-revoke-sub-4f652337-8lc9j\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.636533 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.636512 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f652337-8lc9j\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.681728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.681695 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" Nov 02 22:19:26.682088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.682054 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz" event={"ID":"356ea8f5-7786-44c5-90c3-064b8dd2c626","Type":"ContainerDied","Data":"a8629d10ffa6bca7ca217ca662dea9897695253211ef16b1e78d12916947f311"} Nov 02 22:19:26.682207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.682117 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8629d10ffa6bca7ca217ca662dea9897695253211ef16b1e78d12916947f311" Nov 02 22:19:26.738154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.738122 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqcn\" (UniqueName: \"kubernetes.io/projected/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-kube-api-access-zkqcn\") pod \"maas-api-revoke-sub-4f652337-8lc9j\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.738452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.738430 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f652337-8lc9j\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.738591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.738573 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f652337-8lc9j\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.739202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.739177 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f652337-8lc9j\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.741282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.740737 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f652337-8lc9j\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.747504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.747476 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqcn\" (UniqueName: \"kubernetes.io/projected/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-kube-api-access-zkqcn\") pod \"maas-api-revoke-sub-4f652337-8lc9j\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.888270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.888227 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:26.957539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.957506 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr"] Nov 02 22:19:27.008698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:26.987259 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb"] Nov 02 22:19:27.268595 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:27.268563 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j"] Nov 02 22:19:27.269017 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:19:27.268994 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b5d974_cf70_4c0b_8b0f_86ef31c3c7db.slice/crio-4edaa7781450e83ced8d955388b8e67f1a0f2f480c9e08817c5d9eeb924f6d26 WatchSource:0}: Error finding container 4edaa7781450e83ced8d955388b8e67f1a0f2f480c9e08817c5d9eeb924f6d26: Status 404 returned error can't find the container with id 4edaa7781450e83ced8d955388b8e67f1a0f2f480c9e08817c5d9eeb924f6d26 Nov 02 22:19:27.696647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:27.696481 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" event={"ID":"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db","Type":"ContainerStarted","Data":"4edaa7781450e83ced8d955388b8e67f1a0f2f480c9e08817c5d9eeb924f6d26"} Nov 02 22:19:28.707345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:28.707256 4309 generic.go:356] "Generic (PLEG): container finished" podID="d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db" containerID="3e945989d54efdeee2da835d05379f8cb1ea322b4178cd412f3dc978778ddc67" exitCode=0 Nov 02 22:19:28.707868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:28.707347 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" event={"ID":"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db","Type":"ContainerDied","Data":"3e945989d54efdeee2da835d05379f8cb1ea322b4178cd412f3dc978778ddc67"} Nov 02 22:19:28.709586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:28.709562 4309 generic.go:356] "Generic (PLEG): container finished" podID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerID="dcfc309ca2fa804520d026dd426d75cd01e10f5e03a76b6d8f0e4b88008c6050" exitCode=0 Nov 02 22:19:28.709728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:28.709653 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" event={"ID":"a5748639-a17c-4ea1-9400-ac7fd3519cc7","Type":"ContainerDied","Data":"dcfc309ca2fa804520d026dd426d75cd01e10f5e03a76b6d8f0e4b88008c6050"} Nov 02 22:19:28.710265 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:28.710247 4309 scope.go:122] "RemoveContainer" containerID="dcfc309ca2fa804520d026dd426d75cd01e10f5e03a76b6d8f0e4b88008c6050" Nov 02 22:19:28.712517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:28.712466 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:19:28.712665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:28.712561 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:19:28.712665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:28.712611 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:19:29.224714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.223636 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642"] Nov 02 22:19:29.234695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.233124 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.243262 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.242067 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-419f68-openshift-default-dockercfg-8mkj5\"" Nov 02 22:19:29.247151 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.244227 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642"] Nov 02 22:19:29.310771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.310725 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c8d49-94e7-402f-945c-2888cf3a8930-istiod-ca-cert\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.311137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.311112 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svhs\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-kube-api-access-8svhs\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.311398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.311379 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-token\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.311581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.311563 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-socket\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.311770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.311751 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-certs\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.312102 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.312078 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-credential-socket\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.312190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.312140 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-data\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.312255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.312193 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-envoy\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.312311 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.312284 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-podinfo\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.417695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.415949 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c8d49-94e7-402f-945c-2888cf3a8930-istiod-ca-cert\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.417695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.416051 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svhs\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-kube-api-access-8svhs\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.417695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.416200 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-token\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.417695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.416291 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-socket\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.417695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.416327 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-certs\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.417695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.416419 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-credential-socket\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.417695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.416469 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-data\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.417695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.416527 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-envoy\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.417695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.416628 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-podinfo\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.431713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.429320 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-podinfo\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.431713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.430036 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c8d49-94e7-402f-945c-2888cf3a8930-istiod-ca-cert\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.431713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.430901 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-socket\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.431713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.431147 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-certs\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.445698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.438079 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-data\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.463695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.456284 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-credential-socket\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.463695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.456478 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svhs\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-kube-api-access-8svhs\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.463695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.458474 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-envoy\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.478625 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.478182 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-token\") pod \"e2e-inf-a-419f68-openshift-default-7fbf675875-bv642\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.561323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.561025 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:29.744755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:29.744255 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" event={"ID":"a5748639-a17c-4ea1-9400-ac7fd3519cc7","Type":"ContainerStarted","Data":"521b28391ffab3c26b343332c1879d98e78786f7b001e0cf312d16a99077d4d7"} Nov 02 22:19:30.266583 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.266542 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:19:30.266883 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.266855 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:19:30.267064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.267042 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:19:30.268795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.268341 4309 patch_prober.go:29] interesting pod/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 02 22:19:30.268795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.268380 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 02 22:19:30.280215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.280187 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:30.322712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.322658 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642"] Nov 02 22:19:30.322908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.322757 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:19:30.372726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.372613 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-xks-ca\") pod \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " Nov 02 22:19:30.372881 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.372731 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-kube-api-access-zkqcn\" (UniqueName: \"kubernetes.io/projected/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-kube-api-access-zkqcn\") pod \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " Nov 02 22:19:30.373055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.373030 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-openshift-ca\") pod \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\" (UID: \"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db\") " Nov 02 22:19:30.379699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.378940 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-openshift-ca" pod "d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db" (UID: "d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:30.379699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.379244 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-xks-ca" pod "d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db" (UID: "d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:30.381484 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.381457 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-kube-api-access-zkqcn" pod "d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db" (UID: "d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db"). InnerVolumeSpecName "kube-api-access-zkqcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:30.422435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.422397 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5"] Nov 02 22:19:30.422948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.422919 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" containerID="cri-o://e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6" gracePeriod=30 Nov 02 22:19:30.484259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.484226 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:30.484431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.484412 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqcn\" (UniqueName: \"kubernetes.io/projected/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-kube-api-access-zkqcn\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:30.484540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.484527 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:30.761226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.761185 4309 generic.go:356] "Generic (PLEG): container finished" podID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerID="354ca6fccc191fa28de211107c89c89ec522b9386ce90fdf8f5aa6fd55896571" exitCode=0 Nov 02 22:19:30.762382 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.761278 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" event={"ID":"3d6666a6-9da1-4eca-9b73-2872991b77cc","Type":"ContainerDied","Data":"354ca6fccc191fa28de211107c89c89ec522b9386ce90fdf8f5aa6fd55896571"} Nov 02 22:19:30.763469 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.763425 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" event={"ID":"d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db","Type":"ContainerDied","Data":"4edaa7781450e83ced8d955388b8e67f1a0f2f480c9e08817c5d9eeb924f6d26"} Nov 02 22:19:30.763643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.763476 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4edaa7781450e83ced8d955388b8e67f1a0f2f480c9e08817c5d9eeb924f6d26" Nov 02 22:19:30.763643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.763549 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j" Nov 02 22:19:30.769900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.769800 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" event={"ID":"8e3c8d49-94e7-402f-945c-2888cf3a8930","Type":"ContainerStarted","Data":"1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962"} Nov 02 22:19:30.769900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.769860 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" event={"ID":"8e3c8d49-94e7-402f-945c-2888cf3a8930","Type":"ContainerStarted","Data":"aaf68813b2ebab7f7e32b8ff66bf4133c02eefe9493e192d5f7eff22bc80561d"} Nov 02 22:19:30.829242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:30.829183 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podStartSLOduration=1.829154688 podStartE2EDuration="1.829154688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:30.823568489 +0000 UTC m=+1298.761681708" watchObservedRunningTime="2026-11-02 22:19:30.829154688 +0000 UTC m=+1298.767267909" Nov 02 22:19:31.090707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.090117 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:19:31.177203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.177159 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-data\") pod \"3d6666a6-9da1-4eca-9b73-2872991b77cc\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " Nov 02 22:19:31.177401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.177248 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-kube-api-access-jl5w6\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-kube-api-access-jl5w6\") pod \"3d6666a6-9da1-4eca-9b73-2872991b77cc\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " Nov 02 22:19:31.177401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.177347 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-token\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-token\") pod \"3d6666a6-9da1-4eca-9b73-2872991b77cc\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " Nov 02 22:19:31.177401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.177395 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-credential-socket\") pod \"3d6666a6-9da1-4eca-9b73-2872991b77cc\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " Nov 02 22:19:31.177700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.177473 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-certs\") pod \"3d6666a6-9da1-4eca-9b73-2872991b77cc\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " Nov 02 22:19:31.177700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.177579 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-podinfo\") pod \"3d6666a6-9da1-4eca-9b73-2872991b77cc\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " Nov 02 22:19:31.177700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.177614 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-socket\") pod \"3d6666a6-9da1-4eca-9b73-2872991b77cc\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " Nov 02 22:19:31.177885 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.177794 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6666a6-9da1-4eca-9b73-2872991b77cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d6666a6-9da1-4eca-9b73-2872991b77cc-istiod-ca-cert\") pod \"3d6666a6-9da1-4eca-9b73-2872991b77cc\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " Nov 02 22:19:31.177885 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.177834 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-envoy\") pod \"3d6666a6-9da1-4eca-9b73-2872991b77cc\" (UID: \"3d6666a6-9da1-4eca-9b73-2872991b77cc\") " Nov 02 22:19:31.179320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.179295 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-credential-socket" pod "3d6666a6-9da1-4eca-9b73-2872991b77cc" (UID: "3d6666a6-9da1-4eca-9b73-2872991b77cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:31.179494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.179377 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-data" pod "3d6666a6-9da1-4eca-9b73-2872991b77cc" (UID: "3d6666a6-9da1-4eca-9b73-2872991b77cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:31.180174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.180141 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6666a6-9da1-4eca-9b73-2872991b77cc-istiod-ca-cert" pod "3d6666a6-9da1-4eca-9b73-2872991b77cc" (UID: "3d6666a6-9da1-4eca-9b73-2872991b77cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:31.180438 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.180413 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-socket" pod "3d6666a6-9da1-4eca-9b73-2872991b77cc" (UID: "3d6666a6-9da1-4eca-9b73-2872991b77cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:31.195753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.182307 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-certs" pod "3d6666a6-9da1-4eca-9b73-2872991b77cc" (UID: "3d6666a6-9da1-4eca-9b73-2872991b77cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:31.196402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.196351 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-envoy" pod "3d6666a6-9da1-4eca-9b73-2872991b77cc" (UID: "3d6666a6-9da1-4eca-9b73-2872991b77cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:31.218931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.218412 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-podinfo" pod "3d6666a6-9da1-4eca-9b73-2872991b77cc" (UID: "3d6666a6-9da1-4eca-9b73-2872991b77cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:19:31.218931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.218596 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-kube-api-access-jl5w6" pod "3d6666a6-9da1-4eca-9b73-2872991b77cc" (UID: "3d6666a6-9da1-4eca-9b73-2872991b77cc"). InnerVolumeSpecName "kube-api-access-jl5w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:31.218931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.218811 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-token" pod "3d6666a6-9da1-4eca-9b73-2872991b77cc" (UID: "3d6666a6-9da1-4eca-9b73-2872991b77cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:31.269463 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.269362 4309 patch_prober.go:29] interesting pod/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 02 22:19:31.269635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.269467 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 02 22:19:31.293698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.280737 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:31.293698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.280766 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:31.293698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.280783 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:31.293698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.280796 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:31.293698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.280805 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:31.293698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.280822 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d6666a6-9da1-4eca-9b73-2872991b77cc-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:31.293698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.280852 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:31.293698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.280879 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d6666a6-9da1-4eca-9b73-2872991b77cc-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:31.293698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.280895 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl5w6\" (UniqueName: \"kubernetes.io/projected/3d6666a6-9da1-4eca-9b73-2872991b77cc-kube-api-access-jl5w6\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:31.563648 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.563604 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:31.578245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.578212 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:31.781355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.781325 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" Nov 02 22:19:31.781869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.781359 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj" event={"ID":"3d6666a6-9da1-4eca-9b73-2872991b77cc","Type":"ContainerDied","Data":"7804a6c8328991dd4307c0bfd3519fda892bc63f9c1688981a8791587c170388"} Nov 02 22:19:31.781869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.781417 4309 scope.go:122] "RemoveContainer" containerID="354ca6fccc191fa28de211107c89c89ec522b9386ce90fdf8f5aa6fd55896571" Nov 02 22:19:31.785697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.782342 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:31.799704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.799479 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:19:31.854439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.854407 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj"] Nov 02 22:19:31.869960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:31.869933 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-84a94060-openshift-default-846cfc5c5c-dbsvj"] Nov 02 22:19:32.220110 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:32.220032 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" path="/var/lib/kubelet/pods/3d6666a6-9da1-4eca-9b73-2872991b77cc/volumes" Nov 02 22:19:32.269352 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:32.269316 4309 patch_prober.go:29] interesting pod/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 02 22:19:32.269515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:32.269382 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 02 22:19:32.627596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:32.627557 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6"] Nov 02 22:19:33.269253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:33.269216 4309 patch_prober.go:29] interesting pod/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 02 22:19:33.269867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:33.269837 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 02 22:19:34.270907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:34.270876 4309 patch_prober.go:29] interesting pod/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 02 22:19:34.271315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:34.270927 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 02 22:19:35.270759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.269876 4309 patch_prober.go:29] interesting pod/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Nov 02 22:19:35.270759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.269976 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Nov 02 22:19:35.647700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.645981 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:19:35.916636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.916604 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:19:35.989385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.989345 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31ef3244-ef74-4e12-ae01-22a50f373880-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef3244-ef74-4e12-ae01-22a50f373880-istio-podinfo\") pod \"31ef3244-ef74-4e12-ae01-22a50f373880\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " Nov 02 22:19:35.989855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.989827 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-certs\") pod \"31ef3244-ef74-4e12-ae01-22a50f373880\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " Nov 02 22:19:35.990083 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.990065 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ef3244-ef74-4e12-ae01-22a50f373880-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef3244-ef74-4e12-ae01-22a50f373880-istiod-ca-cert\") pod \"31ef3244-ef74-4e12-ae01-22a50f373880\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " Nov 02 22:19:35.990390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.990369 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-data\") pod \"31ef3244-ef74-4e12-ae01-22a50f373880\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " Nov 02 22:19:35.990611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.990594 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-istio-token\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-istio-token\") pod \"31ef3244-ef74-4e12-ae01-22a50f373880\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " Nov 02 22:19:35.991159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.991132 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-socket\") pod \"31ef3244-ef74-4e12-ae01-22a50f373880\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " Nov 02 22:19:35.991500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.991482 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-credential-socket\") pod \"31ef3244-ef74-4e12-ae01-22a50f373880\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " Nov 02 22:19:35.991659 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.991643 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-envoy\") pod \"31ef3244-ef74-4e12-ae01-22a50f373880\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " Nov 02 22:19:35.993413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:35.992593 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-kube-api-access-dzhtm\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-kube-api-access-dzhtm\") pod \"31ef3244-ef74-4e12-ae01-22a50f373880\" (UID: \"31ef3244-ef74-4e12-ae01-22a50f373880\") " Nov 02 22:19:36.000690 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.000648 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-data" pod "31ef3244-ef74-4e12-ae01-22a50f373880" (UID: "31ef3244-ef74-4e12-ae01-22a50f373880"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:36.003106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.002406 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-credential-socket" pod "31ef3244-ef74-4e12-ae01-22a50f373880" (UID: "31ef3244-ef74-4e12-ae01-22a50f373880"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:36.003106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.002612 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-certs" pod "31ef3244-ef74-4e12-ae01-22a50f373880" (UID: "31ef3244-ef74-4e12-ae01-22a50f373880"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:36.003328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.003299 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ef3244-ef74-4e12-ae01-22a50f373880-istiod-ca-cert" pod "31ef3244-ef74-4e12-ae01-22a50f373880" (UID: "31ef3244-ef74-4e12-ae01-22a50f373880"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:36.007210 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.007180 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:36.007210 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.007211 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31ef3244-ef74-4e12-ae01-22a50f373880-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:36.007414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.007226 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:36.007414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.007246 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:36.012728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.011584 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-socket" pod "31ef3244-ef74-4e12-ae01-22a50f373880" (UID: "31ef3244-ef74-4e12-ae01-22a50f373880"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:36.012728 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.011930 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-kube-api-access-dzhtm" pod "31ef3244-ef74-4e12-ae01-22a50f373880" (UID: "31ef3244-ef74-4e12-ae01-22a50f373880"). InnerVolumeSpecName "kube-api-access-dzhtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:36.042382 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.042346 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31ef3244-ef74-4e12-ae01-22a50f373880-istio-podinfo" pod "31ef3244-ef74-4e12-ae01-22a50f373880" (UID: "31ef3244-ef74-4e12-ae01-22a50f373880"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:19:36.042952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.042484 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-istio-token" pod "31ef3244-ef74-4e12-ae01-22a50f373880" (UID: "31ef3244-ef74-4e12-ae01-22a50f373880"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:36.046761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.046498 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-envoy" pod "31ef3244-ef74-4e12-ae01-22a50f373880" (UID: "31ef3244-ef74-4e12-ae01-22a50f373880"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:36.091238 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.091198 4309 generic.go:356] "Generic (PLEG): container finished" podID="31ef3244-ef74-4e12-ae01-22a50f373880" containerID="e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6" exitCode=0 Nov 02 22:19:36.091424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.091348 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" event={"ID":"31ef3244-ef74-4e12-ae01-22a50f373880","Type":"ContainerDied","Data":"e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6"} Nov 02 22:19:36.091486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.091439 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" event={"ID":"31ef3244-ef74-4e12-ae01-22a50f373880","Type":"ContainerDied","Data":"1c0479ecf3365c6bb8667934403ca3d8c16974214e9d920bc12ee50264a7855d"} Nov 02 22:19:36.091486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.091472 4309 scope.go:122] "RemoveContainer" containerID="e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6" Nov 02 22:19:36.091724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.091706 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5" Nov 02 22:19:36.109923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.109892 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31ef3244-ef74-4e12-ae01-22a50f373880-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:36.109923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.109925 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:36.110137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.109940 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:36.110137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.109955 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31ef3244-ef74-4e12-ae01-22a50f373880-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:36.110137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.109968 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzhtm\" (UniqueName: \"kubernetes.io/projected/31ef3244-ef74-4e12-ae01-22a50f373880-kube-api-access-dzhtm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:36.112292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.112268 4309 scope.go:122] "RemoveContainer" containerID="e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6" Nov 02 22:19:36.112611 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:19:36.112586 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6\": container with ID starting with e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6 not found: ID does not exist" containerID="e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6" Nov 02 22:19:36.112749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.112622 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6"} err="failed to get container status \"e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6\": rpc error: code = NotFound desc = could not find container \"e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6\": container with ID starting with e86c634fc2971080bb7f126a1bdd5d369d9ce5dc5610a472b0746d4310e482a6 not found: ID does not exist" Nov 02 22:19:36.143836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.143730 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5"] Nov 02 22:19:36.152160 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.152124 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-k8pm5"] Nov 02 22:19:36.194213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.194180 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" path="/var/lib/kubelet/pods/31ef3244-ef74-4e12-ae01-22a50f373880/volumes" Nov 02 22:19:36.273980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.273781 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:19:36.276288 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:36.276263 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:19:37.034296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.034265 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw"] Nov 02 22:19:37.035157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.035136 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db" containerName="revoke-keys" Nov 02 22:19:37.035157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.035156 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db" containerName="revoke-keys" Nov 02 22:19:37.035346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.035176 4309 cpu_manager.go:414] "containerMap: removing container" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" Nov 02 22:19:37.035346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.035180 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" Nov 02 22:19:37.035346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.035195 4309 cpu_manager.go:414] "containerMap: removing container" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" Nov 02 22:19:37.035346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.035203 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" Nov 02 22:19:37.035544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.035417 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6666a6-9da1-4eca-9b73-2872991b77cc" containerName="istio-proxy" Nov 02 22:19:37.035544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.035430 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ef3244-ef74-4e12-ae01-22a50f373880" containerName="istio-proxy" Nov 02 22:19:37.035544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.035446 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db" containerName="revoke-keys" Nov 02 22:19:37.041567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.041040 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.045583 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.045558 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:19:37.045775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.045749 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:19:37.045923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.045899 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw"] Nov 02 22:19:37.046965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.046835 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:19:37.047760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.047532 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:19:37.146615 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.146577 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kdd5\" (UniqueName: \"kubernetes.io/projected/0fce0df8-64fb-428f-938c-b542d53ec8d8-kube-api-access-6kdd5\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.146951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.146928 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.147085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.147051 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.248978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.248943 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.249207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.249189 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kdd5\" (UniqueName: \"kubernetes.io/projected/0fce0df8-64fb-428f-938c-b542d53ec8d8-kube-api-access-6kdd5\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.249515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.249490 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.250083 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.250062 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.251404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.251382 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.257943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.257919 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kdd5\" (UniqueName: \"kubernetes.io/projected/0fce0df8-64fb-428f-938c-b542d53ec8d8-kube-api-access-6kdd5\") pod \"maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.373610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.373571 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:37.628290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.628161 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" podUID="13f6d309-4012-493e-a01f-c2e3acfbf6a5" containerName="payload-pre-processing" containerID="cri-o://e6079a91d58ed7d37a2eec9e79403ad53e16c26acbbf7c3752dc4d65aa66f6ba" gracePeriod=25 Nov 02 22:19:37.736793 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:19:37.736752 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fce0df8_64fb_428f_938c_b542d53ec8d8.slice/crio-d26aeac25621149ec2f442fa2896f179b200d48c6e525f96a2fe5d63a4b79384 WatchSource:0}: Error finding container d26aeac25621149ec2f442fa2896f179b200d48c6e525f96a2fe5d63a4b79384: Status 404 returned error can't find the container with id d26aeac25621149ec2f442fa2896f179b200d48c6e525f96a2fe5d63a4b79384 Nov 02 22:19:37.737513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:37.737489 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw"] Nov 02 22:19:38.116437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:38.116406 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" event={"ID":"0fce0df8-64fb-428f-938c-b542d53ec8d8","Type":"ContainerStarted","Data":"e4849ba6806dd284a08f83695bd46ce7dce743412c794fd232731d73ef1d0d68"} Nov 02 22:19:38.116580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:38.116470 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" event={"ID":"0fce0df8-64fb-428f-938c-b542d53ec8d8","Type":"ContainerStarted","Data":"d26aeac25621149ec2f442fa2896f179b200d48c6e525f96a2fe5d63a4b79384"} Nov 02 22:19:38.133252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:38.133209 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" podStartSLOduration=1.133195157 podStartE2EDuration="1.133195157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:38.130952886 +0000 UTC m=+1306.069066104" watchObservedRunningTime="2026-11-02 22:19:38.133195157 +0000 UTC m=+1306.071308374" Nov 02 22:19:39.131075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:39.131027 4309 generic.go:356] "Generic (PLEG): container finished" podID="0fce0df8-64fb-428f-938c-b542d53ec8d8" containerID="e4849ba6806dd284a08f83695bd46ce7dce743412c794fd232731d73ef1d0d68" exitCode=0 Nov 02 22:19:39.131600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:39.131272 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" event={"ID":"0fce0df8-64fb-428f-938c-b542d53ec8d8","Type":"ContainerDied","Data":"e4849ba6806dd284a08f83695bd46ce7dce743412c794fd232731d73ef1d0d68"} Nov 02 22:19:40.322520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.322472 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:40.363697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.340925 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fce0df8-64fb-428f-938c-b542d53ec8d8-kube-api-access-6kdd5\" (UniqueName: \"kubernetes.io/projected/0fce0df8-64fb-428f-938c-b542d53ec8d8-kube-api-access-6kdd5\") pod \"0fce0df8-64fb-428f-938c-b542d53ec8d8\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " Nov 02 22:19:40.363697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.341139 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-xks-ca\") pod \"0fce0df8-64fb-428f-938c-b542d53ec8d8\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " Nov 02 22:19:40.363697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.341214 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-openshift-ca\") pod \"0fce0df8-64fb-428f-938c-b542d53ec8d8\" (UID: \"0fce0df8-64fb-428f-938c-b542d53ec8d8\") " Nov 02 22:19:40.363697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.356042 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-openshift-ca" pod "0fce0df8-64fb-428f-938c-b542d53ec8d8" (UID: "0fce0df8-64fb-428f-938c-b542d53ec8d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:40.366607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.366554 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-xks-ca" pod "0fce0df8-64fb-428f-938c-b542d53ec8d8" (UID: "0fce0df8-64fb-428f-938c-b542d53ec8d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:40.404761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.378967 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fce0df8-64fb-428f-938c-b542d53ec8d8-kube-api-access-6kdd5" pod "0fce0df8-64fb-428f-938c-b542d53ec8d8" (UID: "0fce0df8-64fb-428f-938c-b542d53ec8d8"). InnerVolumeSpecName "kube-api-access-6kdd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:40.449688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.449635 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:40.449688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.449684 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce0df8-64fb-428f-938c-b542d53ec8d8-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:40.449958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:40.449700 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kdd5\" (UniqueName: \"kubernetes.io/projected/0fce0df8-64fb-428f-938c-b542d53ec8d8-kube-api-access-6kdd5\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:41.163375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:41.163339 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" Nov 02 22:19:41.163579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:41.163429 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw" event={"ID":"0fce0df8-64fb-428f-938c-b542d53ec8d8","Type":"ContainerDied","Data":"d26aeac25621149ec2f442fa2896f179b200d48c6e525f96a2fe5d63a4b79384"} Nov 02 22:19:41.163579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:41.163529 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d26aeac25621149ec2f442fa2896f179b200d48c6e525f96a2fe5d63a4b79384" Nov 02 22:19:41.607708 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:19:41.607656 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:19:41.864263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:41.864182 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f"] Nov 02 22:19:41.873131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:41.873103 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4"] Nov 02 22:19:41.994227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:41.994145 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" containerID="cri-o://974433e3f8961267ff849e24d6fd83d4ce9fd3547364968b0e5d50d5253b887a" gracePeriod=45 Nov 02 22:19:42.042271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.042230 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" containerID="cri-o://4db96f12fa60a28bba7284987236404ca845ba2cebbef73b0c22c4505dba55d7" gracePeriod=45 Nov 02 22:19:42.176453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.176416 4309 generic.go:356] "Generic (PLEG): container finished" podID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerID="974433e3f8961267ff849e24d6fd83d4ce9fd3547364968b0e5d50d5253b887a" exitCode=0 Nov 02 22:19:42.176737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.176714 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" event={"ID":"07d8db7d-7bfb-4774-b8e7-062e282680ae","Type":"ContainerDied","Data":"974433e3f8961267ff849e24d6fd83d4ce9fd3547364968b0e5d50d5253b887a"} Nov 02 22:19:42.178451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.178425 4309 generic.go:356] "Generic (PLEG): container finished" podID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerID="4db96f12fa60a28bba7284987236404ca845ba2cebbef73b0c22c4505dba55d7" exitCode=0 Nov 02 22:19:42.178581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.178564 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" event={"ID":"b6f206e9-742a-416e-b4aa-1bf1a93c301d","Type":"ContainerDied","Data":"4db96f12fa60a28bba7284987236404ca845ba2cebbef73b0c22c4505dba55d7"} Nov 02 22:19:42.425540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.425170 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:19:42.431822 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.431797 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:19:42.522997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.522963 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-dshm\") pod \"07d8db7d-7bfb-4774-b8e7-062e282680ae\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " Nov 02 22:19:42.523186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523094 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-dshm\") pod \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " Nov 02 22:19:42.523271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523238 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f206e9-742a-416e-b4aa-1bf1a93c301d-kube-api-access-7tp58\" (UniqueName: \"kubernetes.io/projected/b6f206e9-742a-416e-b4aa-1bf1a93c301d-kube-api-access-7tp58\") pod \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " Nov 02 22:19:42.523333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523274 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-home\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-home\") pod \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " Nov 02 22:19:42.523333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523315 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d8db7d-7bfb-4774-b8e7-062e282680ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/07d8db7d-7bfb-4774-b8e7-062e282680ae-tls-certs\") pod \"07d8db7d-7bfb-4774-b8e7-062e282680ae\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " Nov 02 22:19:42.523445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523342 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tls-certs\") pod \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " Nov 02 22:19:42.523547 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523527 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-model-cache\") pod \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " Nov 02 22:19:42.523616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523576 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d8db7d-7bfb-4774-b8e7-062e282680ae-kube-api-access-9n2t7\" (UniqueName: \"kubernetes.io/projected/07d8db7d-7bfb-4774-b8e7-062e282680ae-kube-api-access-9n2t7\") pod \"07d8db7d-7bfb-4774-b8e7-062e282680ae\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " Nov 02 22:19:42.523616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523605 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-home\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-home\") pod \"07d8db7d-7bfb-4774-b8e7-062e282680ae\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " Nov 02 22:19:42.523806 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523779 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-tmp-dir\") pod \"07d8db7d-7bfb-4774-b8e7-062e282680ae\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " Nov 02 22:19:42.523925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523818 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-model-cache\") pod \"07d8db7d-7bfb-4774-b8e7-062e282680ae\" (UID: \"07d8db7d-7bfb-4774-b8e7-062e282680ae\") " Nov 02 22:19:42.523925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.523883 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tmp-dir\") pod \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\" (UID: \"b6f206e9-742a-416e-b4aa-1bf1a93c301d\") " Nov 02 22:19:42.525102 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.525055 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-model-cache" pod "b6f206e9-742a-416e-b4aa-1bf1a93c301d" (UID: "b6f206e9-742a-416e-b4aa-1bf1a93c301d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:42.525195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.525149 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-home" pod "b6f206e9-742a-416e-b4aa-1bf1a93c301d" (UID: "b6f206e9-742a-416e-b4aa-1bf1a93c301d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:42.525926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.525900 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-model-cache" pod "07d8db7d-7bfb-4774-b8e7-062e282680ae" (UID: "07d8db7d-7bfb-4774-b8e7-062e282680ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:42.526010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.525972 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-home" pod "07d8db7d-7bfb-4774-b8e7-062e282680ae" (UID: "07d8db7d-7bfb-4774-b8e7-062e282680ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:42.526112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.526081 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-tmp-dir" pod "07d8db7d-7bfb-4774-b8e7-062e282680ae" (UID: "07d8db7d-7bfb-4774-b8e7-062e282680ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:42.528251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.528217 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f206e9-742a-416e-b4aa-1bf1a93c301d-kube-api-access-7tp58" pod "b6f206e9-742a-416e-b4aa-1bf1a93c301d" (UID: "b6f206e9-742a-416e-b4aa-1bf1a93c301d"). InnerVolumeSpecName "kube-api-access-7tp58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:42.528751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.528726 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tls-certs" pod "b6f206e9-742a-416e-b4aa-1bf1a93c301d" (UID: "b6f206e9-742a-416e-b4aa-1bf1a93c301d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:42.529862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.529839 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tmp-dir" pod "b6f206e9-742a-416e-b4aa-1bf1a93c301d" (UID: "b6f206e9-742a-416e-b4aa-1bf1a93c301d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:42.531983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.531959 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d8db7d-7bfb-4774-b8e7-062e282680ae-kube-api-access-9n2t7" pod "07d8db7d-7bfb-4774-b8e7-062e282680ae" (UID: "07d8db7d-7bfb-4774-b8e7-062e282680ae"). InnerVolumeSpecName "kube-api-access-9n2t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:42.533493 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.533457 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-dshm" pod "b6f206e9-742a-416e-b4aa-1bf1a93c301d" (UID: "b6f206e9-742a-416e-b4aa-1bf1a93c301d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:42.533592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.533513 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-dshm" pod "07d8db7d-7bfb-4774-b8e7-062e282680ae" (UID: "07d8db7d-7bfb-4774-b8e7-062e282680ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:42.534917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.534895 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d8db7d-7bfb-4774-b8e7-062e282680ae-tls-certs" pod "07d8db7d-7bfb-4774-b8e7-062e282680ae" (UID: "07d8db7d-7bfb-4774-b8e7-062e282680ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649696 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649729 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n2t7\" (UniqueName: \"kubernetes.io/projected/07d8db7d-7bfb-4774-b8e7-062e282680ae-kube-api-access-9n2t7\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649783 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649801 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649813 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649825 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649837 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d8db7d-7bfb-4774-b8e7-062e282680ae-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649848 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649859 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tp58\" (UniqueName: \"kubernetes.io/projected/b6f206e9-742a-416e-b4aa-1bf1a93c301d-kube-api-access-7tp58\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649870 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6f206e9-742a-416e-b4aa-1bf1a93c301d-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649883 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d8db7d-7bfb-4774-b8e7-062e282680ae-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:42.654624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:42.649893 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6f206e9-742a-416e-b4aa-1bf1a93c301d-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:43.192743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.192715 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" Nov 02 22:19:43.192743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.192722 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr" event={"ID":"07d8db7d-7bfb-4774-b8e7-062e282680ae","Type":"ContainerDied","Data":"be6bada33e99044272cb3cb328d31ced31b9dc0b3c68cba6638219f5420d8a8a"} Nov 02 22:19:43.192997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.192774 4309 scope.go:122] "RemoveContainer" containerID="974433e3f8961267ff849e24d6fd83d4ce9fd3547364968b0e5d50d5253b887a" Nov 02 22:19:43.194724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.194688 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" Nov 02 22:19:43.194917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.194775 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" event={"ID":"b6f206e9-742a-416e-b4aa-1bf1a93c301d","Type":"ContainerDied","Data":"00b56213cdd8a395d023fa10a1c086a63b0c77b782ac3a9df3193f2a7e7ebabd"} Nov 02 22:19:43.215359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.215338 4309 scope.go:122] "RemoveContainer" containerID="4db96f12fa60a28bba7284987236404ca845ba2cebbef73b0c22c4505dba55d7" Nov 02 22:19:43.232357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.232334 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr"] Nov 02 22:19:43.259696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.256124 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/distinct-2-w1-5f4a2f-kserve-788b75b6c5-ck7wr"] Nov 02 22:19:43.263803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.262924 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb"] Nov 02 22:19:43.286100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.286064 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb"] Nov 02 22:19:43.359007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.358968 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5f4a2f/simulator-w1-5f4a2f-kserve-84f45d8f6-v5bdb" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:43.801410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.801361 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq"] Nov 02 22:19:43.802651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.802628 4309 cpu_manager.go:414] "containerMap: removing container" podUID="0fce0df8-64fb-428f-938c-b542d53ec8d8" containerName="revoke-keys" Nov 02 22:19:43.802769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.802655 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fce0df8-64fb-428f-938c-b542d53ec8d8" containerName="revoke-keys" Nov 02 22:19:43.802769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.802688 4309 cpu_manager.go:414] "containerMap: removing container" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" Nov 02 22:19:43.802769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.802696 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" Nov 02 22:19:43.802769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.802745 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" Nov 02 22:19:43.802769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.802755 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" Nov 02 22:19:43.802967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.802955 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fce0df8-64fb-428f-938c-b542d53ec8d8" containerName="revoke-keys" Nov 02 22:19:43.803003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.802972 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" containerName="main" Nov 02 22:19:43.803003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.802984 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" containerName="main" Nov 02 22:19:43.818889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.818582 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w"] Nov 02 22:19:43.819638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.819612 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:43.823541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.823489 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:19:43.823541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.823506 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:19:43.823747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.823548 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:19:43.823967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.823949 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:19:43.837266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.837246 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq"] Nov 02 22:19:43.837404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.837379 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:43.837500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:43.837487 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w"] Nov 02 22:19:44.142771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.125765 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcl2j\" (UniqueName: \"kubernetes.io/projected/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-kube-api-access-zcl2j\") pod \"maas-api-revoke-sub-eb47675b-jv7pq\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.142771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.125864 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe2d002-k7f7w\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.142771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.126060 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe2d002-k7f7w\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.142771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.126325 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb47675b-jv7pq\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.142771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.126408 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb47675b-jv7pq\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.142771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.126455 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvsn\" (UniqueName: \"kubernetes.io/projected/2e621f94-bf4b-41de-956a-700fa5f5af20-kube-api-access-zwvsn\") pod \"maas-api-revoke-sub-1fe2d002-k7f7w\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.231381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.228498 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d8db7d-7bfb-4774-b8e7-062e282680ae" path="/var/lib/kubelet/pods/07d8db7d-7bfb-4774-b8e7-062e282680ae/volumes" Nov 02 22:19:44.231381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.229141 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb47675b-jv7pq\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.231381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.229196 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f206e9-742a-416e-b4aa-1bf1a93c301d" path="/var/lib/kubelet/pods/b6f206e9-742a-416e-b4aa-1bf1a93c301d/volumes" Nov 02 22:19:44.231381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.229211 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb47675b-jv7pq\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.231381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.229248 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvsn\" (UniqueName: \"kubernetes.io/projected/2e621f94-bf4b-41de-956a-700fa5f5af20-kube-api-access-zwvsn\") pod \"maas-api-revoke-sub-1fe2d002-k7f7w\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.231381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.229336 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcl2j\" (UniqueName: \"kubernetes.io/projected/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-kube-api-access-zcl2j\") pod \"maas-api-revoke-sub-eb47675b-jv7pq\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.231381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.229386 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe2d002-k7f7w\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.231381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.229535 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe2d002-k7f7w\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.231381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.231001 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb47675b-jv7pq\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.234549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.234525 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe2d002-k7f7w\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.235642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.235619 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe2d002-k7f7w\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.237739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.236787 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb47675b-jv7pq\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.240960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.240931 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvsn\" (UniqueName: \"kubernetes.io/projected/2e621f94-bf4b-41de-956a-700fa5f5af20-kube-api-access-zwvsn\") pod \"maas-api-revoke-sub-1fe2d002-k7f7w\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.247235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.247209 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcl2j\" (UniqueName: \"kubernetes.io/projected/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-kube-api-access-zcl2j\") pod \"maas-api-revoke-sub-eb47675b-jv7pq\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.309726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.307940 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq"] Nov 02 22:19:44.313413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.313383 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14dae2-wzxdq"] Nov 02 22:19:44.374369 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.374342 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:44.459309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.459134 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:44.685363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.685336 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w"] Nov 02 22:19:44.686871 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:19:44.686843 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e621f94_bf4b_41de_956a_700fa5f5af20.slice/crio-1ccd091c85fb07c2b1cb0c6613273f84c5bb429bfe32c8c1f65bb65a827cbb8e WatchSource:0}: Error finding container 1ccd091c85fb07c2b1cb0c6613273f84c5bb429bfe32c8c1f65bb65a827cbb8e: Status 404 returned error can't find the container with id 1ccd091c85fb07c2b1cb0c6613273f84c5bb429bfe32c8c1f65bb65a827cbb8e Nov 02 22:19:44.801371 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:19:44.801337 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb41b1fe_7c1f_45da_bd49_a7ca79ea921e.slice/crio-2eae06fd0f3a5fb10b675d5ae6ce7873f93fc0bd599795363f67143e11eaecf6 WatchSource:0}: Error finding container 2eae06fd0f3a5fb10b675d5ae6ce7873f93fc0bd599795363f67143e11eaecf6: Status 404 returned error can't find the container with id 2eae06fd0f3a5fb10b675d5ae6ce7873f93fc0bd599795363f67143e11eaecf6 Nov 02 22:19:44.804021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:44.803992 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq"] Nov 02 22:19:45.216435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:45.216398 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx"] Nov 02 22:19:45.217471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:45.217434 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" containerID="cri-o://4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40" gracePeriod=30 Nov 02 22:19:45.246189 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:45.246154 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" event={"ID":"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e","Type":"ContainerStarted","Data":"2eae06fd0f3a5fb10b675d5ae6ce7873f93fc0bd599795363f67143e11eaecf6"} Nov 02 22:19:45.250882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:45.250850 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" event={"ID":"2e621f94-bf4b-41de-956a-700fa5f5af20","Type":"ContainerStarted","Data":"e6fa27d4cb66a7796f832a3f04abfcfdbadee80d3772ae5db853eb59476c3253"} Nov 02 22:19:45.251098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:45.251070 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" event={"ID":"2e621f94-bf4b-41de-956a-700fa5f5af20","Type":"ContainerStarted","Data":"1ccd091c85fb07c2b1cb0c6613273f84c5bb429bfe32c8c1f65bb65a827cbb8e"} Nov 02 22:19:45.275710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:45.275629 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" podStartSLOduration=2.275615397 podStartE2EDuration="2.275615397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:45.270461206 +0000 UTC m=+1313.208574423" watchObservedRunningTime="2026-11-02 22:19:45.275615397 +0000 UTC m=+1313.213728657" Nov 02 22:19:45.424712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:45.424644 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79"] Nov 02 22:19:45.424898 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:45.424870 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" containerID="cri-o://ec836f011e5b5e48ef6a6b0f6bf6e0304d9d3d07d001f31671cca00c9b964d57" gracePeriod=30 Nov 02 22:19:46.193403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:46.193370 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42cf933-c88e-4a4a-a3f7-74f669f71f8d" path="/var/lib/kubelet/pods/b42cf933-c88e-4a4a-a3f7-74f669f71f8d/volumes" Nov 02 22:19:46.260951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:46.260919 4309 generic.go:356] "Generic (PLEG): container finished" podID="2e621f94-bf4b-41de-956a-700fa5f5af20" containerID="e6fa27d4cb66a7796f832a3f04abfcfdbadee80d3772ae5db853eb59476c3253" exitCode=0 Nov 02 22:19:46.261094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:46.261005 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" event={"ID":"2e621f94-bf4b-41de-956a-700fa5f5af20","Type":"ContainerDied","Data":"e6fa27d4cb66a7796f832a3f04abfcfdbadee80d3772ae5db853eb59476c3253"} Nov 02 22:19:46.262916 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:46.262893 4309 generic.go:356] "Generic (PLEG): container finished" podID="bb41b1fe-7c1f-45da-bd49-a7ca79ea921e" containerID="7a150a935524d0671ee45c697fef1e6b4c19bd1a694fd24e191a0ac049cade77" exitCode=0 Nov 02 22:19:46.263034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:46.262933 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" event={"ID":"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e","Type":"ContainerDied","Data":"7a150a935524d0671ee45c697fef1e6b4c19bd1a694fd24e191a0ac049cade77"} Nov 02 22:19:47.154647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.154606 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" podUID="13f6d309-4012-493e-a01f-c2e3acfbf6a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:19:47.189601 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.189562 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:19:47.189814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.189621 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:19:47.838522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.838493 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:47.948699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.943266 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-openshift-ca\") pod \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " Nov 02 22:19:47.948699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.943369 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-xks-ca\") pod \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " Nov 02 22:19:47.948699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.943843 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-openshift-ca" pod "bb41b1fe-7c1f-45da-bd49-a7ca79ea921e" (UID: "bb41b1fe-7c1f-45da-bd49-a7ca79ea921e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:47.948699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.944016 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-kube-api-access-zcl2j\" (UniqueName: \"kubernetes.io/projected/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-kube-api-access-zcl2j\") pod \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\" (UID: \"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e\") " Nov 02 22:19:47.948699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.944943 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:47.998498 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.998465 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-kube-api-access-zcl2j" pod "bb41b1fe-7c1f-45da-bd49-a7ca79ea921e" (UID: "bb41b1fe-7c1f-45da-bd49-a7ca79ea921e"). InnerVolumeSpecName "kube-api-access-zcl2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:47.999105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:47.999083 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-xks-ca" pod "bb41b1fe-7c1f-45da-bd49-a7ca79ea921e" (UID: "bb41b1fe-7c1f-45da-bd49-a7ca79ea921e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:48.010701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.010658 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:48.045750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.045716 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-openshift-ca\") pod \"2e621f94-bf4b-41de-956a-700fa5f5af20\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " Nov 02 22:19:48.045921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.045842 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-xks-ca\") pod \"2e621f94-bf4b-41de-956a-700fa5f5af20\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " Nov 02 22:19:48.046099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.046072 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-openshift-ca" pod "2e621f94-bf4b-41de-956a-700fa5f5af20" (UID: "2e621f94-bf4b-41de-956a-700fa5f5af20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:48.046240 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.046102 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e621f94-bf4b-41de-956a-700fa5f5af20-kube-api-access-zwvsn\" (UniqueName: \"kubernetes.io/projected/2e621f94-bf4b-41de-956a-700fa5f5af20-kube-api-access-zwvsn\") pod \"2e621f94-bf4b-41de-956a-700fa5f5af20\" (UID: \"2e621f94-bf4b-41de-956a-700fa5f5af20\") " Nov 02 22:19:48.047154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.047126 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:48.047154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.047155 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcl2j\" (UniqueName: \"kubernetes.io/projected/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e-kube-api-access-zcl2j\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:48.047337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.047169 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:48.048410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.048380 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e621f94-bf4b-41de-956a-700fa5f5af20-kube-api-access-zwvsn" pod "2e621f94-bf4b-41de-956a-700fa5f5af20" (UID: "2e621f94-bf4b-41de-956a-700fa5f5af20"). InnerVolumeSpecName "kube-api-access-zwvsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:48.048747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.048720 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-xks-ca" pod "2e621f94-bf4b-41de-956a-700fa5f5af20" (UID: "2e621f94-bf4b-41de-956a-700fa5f5af20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:48.149040 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.148960 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwvsn\" (UniqueName: \"kubernetes.io/projected/2e621f94-bf4b-41de-956a-700fa5f5af20-kube-api-access-zwvsn\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:48.149040 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.148988 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e621f94-bf4b-41de-956a-700fa5f5af20-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:48.296353 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.296324 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" Nov 02 22:19:48.296664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.296355 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq" event={"ID":"bb41b1fe-7c1f-45da-bd49-a7ca79ea921e","Type":"ContainerDied","Data":"2eae06fd0f3a5fb10b675d5ae6ce7873f93fc0bd599795363f67143e11eaecf6"} Nov 02 22:19:48.296814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.296736 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eae06fd0f3a5fb10b675d5ae6ce7873f93fc0bd599795363f67143e11eaecf6" Nov 02 22:19:48.298413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.298391 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" Nov 02 22:19:48.298512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.298426 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w" event={"ID":"2e621f94-bf4b-41de-956a-700fa5f5af20","Type":"ContainerDied","Data":"1ccd091c85fb07c2b1cb0c6613273f84c5bb429bfe32c8c1f65bb65a827cbb8e"} Nov 02 22:19:48.298512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:48.298475 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ccd091c85fb07c2b1cb0c6613273f84c5bb429bfe32c8c1f65bb65a827cbb8e" Nov 02 22:19:50.791544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.791139 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.947002 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9"] Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.949014 4309 cpu_manager.go:414] "containerMap: removing container" podUID="2e621f94-bf4b-41de-956a-700fa5f5af20" containerName="revoke-keys" Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.949036 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e621f94-bf4b-41de-956a-700fa5f5af20" containerName="revoke-keys" Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.949086 4309 cpu_manager.go:414] "containerMap: removing container" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.949095 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.949195 4309 cpu_manager.go:414] "containerMap: removing container" podUID="bb41b1fe-7c1f-45da-bd49-a7ca79ea921e" containerName="revoke-keys" Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.949204 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb41b1fe-7c1f-45da-bd49-a7ca79ea921e" containerName="revoke-keys" Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.949568 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e621f94-bf4b-41de-956a-700fa5f5af20" containerName="revoke-keys" Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.949605 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" containerName="istio-proxy" Nov 02 22:19:50.949802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.949632 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb41b1fe-7c1f-45da-bd49-a7ca79ea921e" containerName="revoke-keys" Nov 02 22:19:50.964900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.964852 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd"] Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.972953 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.976762 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-data\") pod \"4b9ba007-fba2-4721-90a9-b9512043544b\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.976824 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-envoy\") pod \"4b9ba007-fba2-4721-90a9-b9512043544b\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.976879 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-credential-socket\") pod \"4b9ba007-fba2-4721-90a9-b9512043544b\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.977017 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-kube-api-access-cnjwl\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-kube-api-access-cnjwl\") pod \"4b9ba007-fba2-4721-90a9-b9512043544b\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.977213 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-istio-token\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-istio-token\") pod \"4b9ba007-fba2-4721-90a9-b9512043544b\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.977371 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-socket\") pod \"4b9ba007-fba2-4721-90a9-b9512043544b\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.977476 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-certs\") pod \"4b9ba007-fba2-4721-90a9-b9512043544b\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.977567 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b9ba007-fba2-4721-90a9-b9512043544b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9ba007-fba2-4721-90a9-b9512043544b-istio-podinfo\") pod \"4b9ba007-fba2-4721-90a9-b9512043544b\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " Nov 02 22:19:50.977688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.977599 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9ba007-fba2-4721-90a9-b9512043544b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9ba007-fba2-4721-90a9-b9512043544b-istiod-ca-cert\") pod \"4b9ba007-fba2-4721-90a9-b9512043544b\" (UID: \"4b9ba007-fba2-4721-90a9-b9512043544b\") " Nov 02 22:19:50.978362 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.978314 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:19:50.978415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.978373 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f654977b-q2bd9\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:50.978653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.978629 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f654977b-q2bd9\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:50.978762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.978739 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjxt\" (UniqueName: \"kubernetes.io/projected/06c172f3-cb47-4972-b252-908b26ef5616-kube-api-access-xcjxt\") pod \"maas-api-revoke-sub-f654977b-q2bd9\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:50.979231 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.979203 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:19:50.979329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.979280 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:19:50.979390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.979362 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:19:50.979479 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.979454 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-certs" pod "4b9ba007-fba2-4721-90a9-b9512043544b" (UID: "4b9ba007-fba2-4721-90a9-b9512043544b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:50.979737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.979642 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-socket" pod "4b9ba007-fba2-4721-90a9-b9512043544b" (UID: "4b9ba007-fba2-4721-90a9-b9512043544b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:50.979899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.979742 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9ba007-fba2-4721-90a9-b9512043544b-istiod-ca-cert" pod "4b9ba007-fba2-4721-90a9-b9512043544b" (UID: "4b9ba007-fba2-4721-90a9-b9512043544b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:50.980022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.979998 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-credential-socket" pod "4b9ba007-fba2-4721-90a9-b9512043544b" (UID: "4b9ba007-fba2-4721-90a9-b9512043544b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:50.980410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:50.980346 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-data" pod "4b9ba007-fba2-4721-90a9-b9512043544b" (UID: "4b9ba007-fba2-4721-90a9-b9512043544b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:51.014995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.014962 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-envoy" pod "4b9ba007-fba2-4721-90a9-b9512043544b" (UID: "4b9ba007-fba2-4721-90a9-b9512043544b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:51.018944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.018794 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-istio-token" pod "4b9ba007-fba2-4721-90a9-b9512043544b" (UID: "4b9ba007-fba2-4721-90a9-b9512043544b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:51.019829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.019723 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b9ba007-fba2-4721-90a9-b9512043544b-istio-podinfo" pod "4b9ba007-fba2-4721-90a9-b9512043544b" (UID: "4b9ba007-fba2-4721-90a9-b9512043544b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:19:51.074251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.070743 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd"] Nov 02 22:19:51.074251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.070820 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9"] Nov 02 22:19:51.074251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.070980 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.143258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.103776 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-kube-api-access-cnjwl" pod "4b9ba007-fba2-4721-90a9-b9512043544b" (UID: "4b9ba007-fba2-4721-90a9-b9512043544b"). InnerVolumeSpecName "kube-api-access-cnjwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:51.143258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.120092 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f654977b-q2bd9\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:51.143258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.120234 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2051633e-5m2kd\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.143258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.120335 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2051633e-5m2kd\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.143258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.120411 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f654977b-q2bd9\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:51.143258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.120498 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjxt\" (UniqueName: \"kubernetes.io/projected/06c172f3-cb47-4972-b252-908b26ef5616-kube-api-access-xcjxt\") pod \"maas-api-revoke-sub-f654977b-q2bd9\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:51.143258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.121421 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f654977b-q2bd9\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:51.143258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.141238 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f654977b-q2bd9\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.120665 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24kn5\" (UniqueName: \"kubernetes.io/projected/b5d81db8-ebfe-4ec8-80f6-b93face79b11-kube-api-access-24kn5\") pod \"maas-api-revoke-sub-2051633e-5m2kd\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.170444 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.170468 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.170485 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.170500 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b9ba007-fba2-4721-90a9-b9512043544b-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.170514 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b9ba007-fba2-4721-90a9-b9512043544b-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.170529 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.170546 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.170560 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b9ba007-fba2-4721-90a9-b9512043544b-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.171000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.170575 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnjwl\" (UniqueName: \"kubernetes.io/projected/4b9ba007-fba2-4721-90a9-b9512043544b-kube-api-access-cnjwl\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.183826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.183794 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjxt\" (UniqueName: \"kubernetes.io/projected/06c172f3-cb47-4972-b252-908b26ef5616-kube-api-access-xcjxt\") pod \"maas-api-revoke-sub-f654977b-q2bd9\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:51.219738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.219280 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:51.276178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.276130 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24kn5\" (UniqueName: \"kubernetes.io/projected/b5d81db8-ebfe-4ec8-80f6-b93face79b11-kube-api-access-24kn5\") pod \"maas-api-revoke-sub-2051633e-5m2kd\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.276741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.276713 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2051633e-5m2kd\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.276958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.276938 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2051633e-5m2kd\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.293027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.282511 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2051633e-5m2kd\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.301949 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.301889 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2051633e-5m2kd\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.304538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.304483 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24kn5\" (UniqueName: \"kubernetes.io/projected/b5d81db8-ebfe-4ec8-80f6-b93face79b11-kube-api-access-24kn5\") pod \"maas-api-revoke-sub-2051633e-5m2kd\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.415050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.415012 4309 generic.go:356] "Generic (PLEG): container finished" podID="4b9ba007-fba2-4721-90a9-b9512043544b" containerID="4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40" exitCode=0 Nov 02 22:19:51.415273 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.415250 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" event={"ID":"4b9ba007-fba2-4721-90a9-b9512043544b","Type":"ContainerDied","Data":"4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40"} Nov 02 22:19:51.415359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.415332 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" event={"ID":"4b9ba007-fba2-4721-90a9-b9512043544b","Type":"ContainerDied","Data":"c392baf505cc72c3e72bb71131847f3c62890a438ab6f44b2ee5a3f9160a408b"} Nov 02 22:19:51.415359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.415338 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx" Nov 02 22:19:51.415496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.415387 4309 scope.go:122] "RemoveContainer" containerID="4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40" Nov 02 22:19:51.424685 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.424644 4309 generic.go:356] "Generic (PLEG): container finished" podID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerID="ec836f011e5b5e48ef6a6b0f6bf6e0304d9d3d07d001f31671cca00c9b964d57" exitCode=0 Nov 02 22:19:51.424810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.424788 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" event={"ID":"28d03196-1dc0-4900-a4a4-a85e289badd4","Type":"ContainerDied","Data":"ec836f011e5b5e48ef6a6b0f6bf6e0304d9d3d07d001f31671cca00c9b964d57"} Nov 02 22:19:51.429424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.429401 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:19:51.445161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.445140 4309 scope.go:122] "RemoveContainer" containerID="4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40" Nov 02 22:19:51.445639 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:19:51.445608 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40\": container with ID starting with 4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40 not found: ID does not exist" containerID="4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40" Nov 02 22:19:51.445773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.445652 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40"} err="failed to get container status \"4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40\": rpc error: code = NotFound desc = could not find container \"4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40\": container with ID starting with 4686b4500b9164f663ad0b9d99c90b1509bd5d5db5d080d28504a75b077e6e40 not found: ID does not exist" Nov 02 22:19:51.447749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.447730 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx"] Nov 02 22:19:51.470093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.469469 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-45dbx"] Nov 02 22:19:51.505583 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.505540 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-credential-socket\") pod \"28d03196-1dc0-4900-a4a4-a85e289badd4\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " Nov 02 22:19:51.505807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.505711 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-certs\") pod \"28d03196-1dc0-4900-a4a4-a85e289badd4\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " Nov 02 22:19:51.505892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.505880 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-socket\") pod \"28d03196-1dc0-4900-a4a4-a85e289badd4\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " Nov 02 22:19:51.506032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.505992 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d03196-1dc0-4900-a4a4-a85e289badd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d03196-1dc0-4900-a4a4-a85e289badd4-istiod-ca-cert\") pod \"28d03196-1dc0-4900-a4a4-a85e289badd4\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " Nov 02 22:19:51.506107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.506046 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-data\") pod \"28d03196-1dc0-4900-a4a4-a85e289badd4\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " Nov 02 22:19:51.506107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.506081 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-kube-api-access-28244\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-kube-api-access-28244\") pod \"28d03196-1dc0-4900-a4a4-a85e289badd4\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " Nov 02 22:19:51.506216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.506146 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-podinfo\") pod \"28d03196-1dc0-4900-a4a4-a85e289badd4\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " Nov 02 22:19:51.506271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.506226 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-token\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-token\") pod \"28d03196-1dc0-4900-a4a4-a85e289badd4\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " Nov 02 22:19:51.506271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.506217 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-credential-socket" pod "28d03196-1dc0-4900-a4a4-a85e289badd4" (UID: "28d03196-1dc0-4900-a4a4-a85e289badd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:51.506271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.506261 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-envoy\") pod \"28d03196-1dc0-4900-a4a4-a85e289badd4\" (UID: \"28d03196-1dc0-4900-a4a4-a85e289badd4\") " Nov 02 22:19:51.506432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.506280 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-socket" pod "28d03196-1dc0-4900-a4a4-a85e289badd4" (UID: "28d03196-1dc0-4900-a4a4-a85e289badd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:51.507129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.506971 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d03196-1dc0-4900-a4a4-a85e289badd4-istiod-ca-cert" pod "28d03196-1dc0-4900-a4a4-a85e289badd4" (UID: "28d03196-1dc0-4900-a4a4-a85e289badd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:51.507214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.507138 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.507214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.507164 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28d03196-1dc0-4900-a4a4-a85e289badd4-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.507214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.507180 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.507430 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.507394 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-data" pod "28d03196-1dc0-4900-a4a4-a85e289badd4" (UID: "28d03196-1dc0-4900-a4a4-a85e289badd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:51.507549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.507526 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-certs" pod "28d03196-1dc0-4900-a4a4-a85e289badd4" (UID: "28d03196-1dc0-4900-a4a4-a85e289badd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:51.509760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.509651 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-kube-api-access-28244" pod "28d03196-1dc0-4900-a4a4-a85e289badd4" (UID: "28d03196-1dc0-4900-a4a4-a85e289badd4"). InnerVolumeSpecName "kube-api-access-28244". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:51.510384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.510360 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-podinfo" pod "28d03196-1dc0-4900-a4a4-a85e289badd4" (UID: "28d03196-1dc0-4900-a4a4-a85e289badd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:19:51.510638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.510617 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-token" pod "28d03196-1dc0-4900-a4a4-a85e289badd4" (UID: "28d03196-1dc0-4900-a4a4-a85e289badd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:51.517824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.517792 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-envoy" pod "28d03196-1dc0-4900-a4a4-a85e289badd4" (UID: "28d03196-1dc0-4900-a4a4-a85e289badd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:51.554797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.554769 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:51.611411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.610345 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.611411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.610373 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28244\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-kube-api-access-28244\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.611411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.610382 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.611411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.610392 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.611411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.610402 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.611411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.610412 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28d03196-1dc0-4900-a4a4-a85e289badd4-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:51.685036 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.684970 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9"] Nov 02 22:19:51.685940 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:19:51.685911 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c172f3_cb47_4972_b252_908b26ef5616.slice/crio-5d01e105281aa6715217e5b17ceade0541216429b25a80da68b6d9abd55111ae WatchSource:0}: Error finding container 5d01e105281aa6715217e5b17ceade0541216429b25a80da68b6d9abd55111ae: Status 404 returned error can't find the container with id 5d01e105281aa6715217e5b17ceade0541216429b25a80da68b6d9abd55111ae Nov 02 22:19:51.895043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:51.894994 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd"] Nov 02 22:19:51.924483 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:19:51.924455 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d81db8_ebfe_4ec8_80f6_b93face79b11.slice/crio-3094971b23b6d0b3a4accbdbcbde51a88225920f7d9ab5691c759fdf07e3ca1c WatchSource:0}: Error finding container 3094971b23b6d0b3a4accbdbcbde51a88225920f7d9ab5691c759fdf07e3ca1c: Status 404 returned error can't find the container with id 3094971b23b6d0b3a4accbdbcbde51a88225920f7d9ab5691c759fdf07e3ca1c Nov 02 22:19:52.196764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.196638 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9ba007-fba2-4721-90a9-b9512043544b" path="/var/lib/kubelet/pods/4b9ba007-fba2-4721-90a9-b9512043544b/volumes" Nov 02 22:19:52.518207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.518112 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" event={"ID":"b5d81db8-ebfe-4ec8-80f6-b93face79b11","Type":"ContainerStarted","Data":"3094971b23b6d0b3a4accbdbcbde51a88225920f7d9ab5691c759fdf07e3ca1c"} Nov 02 22:19:52.531720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.531687 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" event={"ID":"28d03196-1dc0-4900-a4a4-a85e289badd4","Type":"ContainerDied","Data":"782d58080d2c62e21ac8a900dd7599aa4c86bd3c6a7bd805b183f6812e7c316d"} Nov 02 22:19:52.531941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.531922 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79" Nov 02 22:19:52.532056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.532042 4309 scope.go:122] "RemoveContainer" containerID="ec836f011e5b5e48ef6a6b0f6bf6e0304d9d3d07d001f31671cca00c9b964d57" Nov 02 22:19:52.539667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.539640 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" event={"ID":"06c172f3-cb47-4972-b252-908b26ef5616","Type":"ContainerStarted","Data":"a135876985cc8feaef3f231309107e2abd066b3146ab689985aa77d0f4d6ad2c"} Nov 02 22:19:52.539858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.539837 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" event={"ID":"06c172f3-cb47-4972-b252-908b26ef5616","Type":"ContainerStarted","Data":"5d01e105281aa6715217e5b17ceade0541216429b25a80da68b6d9abd55111ae"} Nov 02 22:19:52.593283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.592955 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79"] Nov 02 22:19:52.633172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.631812 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-c4g79"] Nov 02 22:19:52.640227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:52.638839 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" podStartSLOduration=2.638819614 podStartE2EDuration="2.638819614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:52.615158225 +0000 UTC m=+1320.553271444" watchObservedRunningTime="2026-11-02 22:19:52.638819614 +0000 UTC m=+1320.576932833" Nov 02 22:19:53.619702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:53.619434 4309 generic.go:356] "Generic (PLEG): container finished" podID="b5d81db8-ebfe-4ec8-80f6-b93face79b11" containerID="743522f113eca95a76d09e2a4145f2c633911071646ac934e73d2193289f09b4" exitCode=0 Nov 02 22:19:53.619702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:53.619574 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" event={"ID":"b5d81db8-ebfe-4ec8-80f6-b93face79b11","Type":"ContainerDied","Data":"743522f113eca95a76d09e2a4145f2c633911071646ac934e73d2193289f09b4"} Nov 02 22:19:53.636691 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:53.635901 4309 generic.go:356] "Generic (PLEG): container finished" podID="06c172f3-cb47-4972-b252-908b26ef5616" containerID="a135876985cc8feaef3f231309107e2abd066b3146ab689985aa77d0f4d6ad2c" exitCode=0 Nov 02 22:19:53.636691 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:53.636097 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" event={"ID":"06c172f3-cb47-4972-b252-908b26ef5616","Type":"ContainerDied","Data":"a135876985cc8feaef3f231309107e2abd066b3146ab689985aa77d0f4d6ad2c"} Nov 02 22:19:54.240738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.238754 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" path="/var/lib/kubelet/pods/28d03196-1dc0-4900-a4a4-a85e289badd4/volumes" Nov 02 22:19:54.549758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.539736 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft"] Nov 02 22:19:54.549758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.542041 4309 cpu_manager.go:414] "containerMap: removing container" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" Nov 02 22:19:54.549758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.542066 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" Nov 02 22:19:54.549758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.542505 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d03196-1dc0-4900-a4a4-a85e289badd4" containerName="istio-proxy" Nov 02 22:19:54.848834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.822217 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.855918 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.855885 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a54217-openshift-default-dockercfg-7tr9f\"" Nov 02 22:19:54.892620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.891323 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft"] Nov 02 22:19:54.892816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.892697 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.892816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.892763 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istiod-ca-cert\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.892816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.892795 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-envoy\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.892998 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.892848 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-podinfo\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.892998 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.892934 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-certs\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.892998 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.892965 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-credential-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.893163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.893031 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-token\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.893163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.893144 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtv5\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-kube-api-access-nwtv5\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.894783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.893316 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-data\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:54.968891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:54.958950 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.029124 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.029185 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istiod-ca-cert\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.029216 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-envoy\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.029261 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-podinfo\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.029333 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-certs\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.029363 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-credential-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.029425 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-token\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.029523 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtv5\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-kube-api-access-nwtv5\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.029694 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-data\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.039154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.030184 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-data\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.063697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.050109 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-credential-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.064275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.064240 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-certs\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.070800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.065113 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istiod-ca-cert\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.075697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.071276 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-socket\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.085801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.085758 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:55.148746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.146382 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtv5\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-kube-api-access-nwtv5\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.222697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.199961 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-envoy\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.352703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.277778 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-token\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.353600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.353543 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-podinfo\") pod \"e2e-inf-a-a54217-openshift-default-554d688f75-qlbft\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.567791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.566775 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:55.611715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.586968 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:55.684153 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.656838 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:55.736887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:55.736852 4309 scope.go:122] "RemoveContainer" containerID="fbc504eda0a6bcf369599b870d1a7a8c9e7de2178819633571e0fd2ce4b98972" Nov 02 22:19:56.124699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.124228 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:56.125179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.124755 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:19:56.300429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.300332 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 22:19:56.420697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.413016 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:56.522134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.522085 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:56.661490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.661451 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:56.662446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.662404 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:19:56.822603 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.821570 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:19:57.004189 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.867258 4309 scope.go:122] "RemoveContainer" containerID="ffc7844bdaf765fad3247bc25b0c64a0a7be1d82499054109f01f535e241d2d1" Nov 02 22:19:57.004189 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:56.962497 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" containerID="cri-o://ba56afd4a07a32460fa25e45ec27eca73d5b4abaa7e86147a713cdce1103c9e3" gracePeriod=45 Nov 02 22:19:57.082359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.081853 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:19:57.103211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.103181 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:19:57.150412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.150367 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" containerID="cri-o://e90610af491d0bd067b1790b5bb74e9514e7e2d83cb273efdf8df10b04e08c38" gracePeriod=45 Nov 02 22:19:57.321471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.321367 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" podUID="13f6d309-4012-493e-a01f-c2e3acfbf6a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:19:57.327772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.327746 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:57.340332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.340307 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:57.400052 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.400008 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused" Nov 02 22:19:57.400623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.400512 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": dial tcp 10.134.0.76:8000: connect: connection refused" Nov 02 22:19:57.433941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.415744 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d81db8-ebfe-4ec8-80f6-b93face79b11-kube-api-access-24kn5\" (UniqueName: \"kubernetes.io/projected/b5d81db8-ebfe-4ec8-80f6-b93face79b11-kube-api-access-24kn5\") pod \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " Nov 02 22:19:57.433941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.415905 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-openshift-ca\") pod \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " Nov 02 22:19:57.433941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.415947 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-xks-ca\") pod \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\" (UID: \"b5d81db8-ebfe-4ec8-80f6-b93face79b11\") " Nov 02 22:19:57.433941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.415988 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c172f3-cb47-4972-b252-908b26ef5616-kube-api-access-xcjxt\" (UniqueName: \"kubernetes.io/projected/06c172f3-cb47-4972-b252-908b26ef5616-kube-api-access-xcjxt\") pod \"06c172f3-cb47-4972-b252-908b26ef5616\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " Nov 02 22:19:57.433941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.416097 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-xks-ca\") pod \"06c172f3-cb47-4972-b252-908b26ef5616\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " Nov 02 22:19:57.433941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.416344 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-openshift-ca\") pod \"06c172f3-cb47-4972-b252-908b26ef5616\" (UID: \"06c172f3-cb47-4972-b252-908b26ef5616\") " Nov 02 22:19:57.433941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.423382 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-openshift-ca" pod "b5d81db8-ebfe-4ec8-80f6-b93face79b11" (UID: "b5d81db8-ebfe-4ec8-80f6-b93face79b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:57.453024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.451002 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-openshift-ca" pod "06c172f3-cb47-4972-b252-908b26ef5616" (UID: "06c172f3-cb47-4972-b252-908b26ef5616"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:57.492010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.491930 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c172f3-cb47-4972-b252-908b26ef5616-kube-api-access-xcjxt" pod "06c172f3-cb47-4972-b252-908b26ef5616" (UID: "06c172f3-cb47-4972-b252-908b26ef5616"). InnerVolumeSpecName "kube-api-access-xcjxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:57.492344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.492304 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d81db8-ebfe-4ec8-80f6-b93face79b11-kube-api-access-24kn5" pod "b5d81db8-ebfe-4ec8-80f6-b93face79b11" (UID: "b5d81db8-ebfe-4ec8-80f6-b93face79b11"). InnerVolumeSpecName "kube-api-access-24kn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:57.492664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.492624 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-xks-ca" pod "b5d81db8-ebfe-4ec8-80f6-b93face79b11" (UID: "b5d81db8-ebfe-4ec8-80f6-b93face79b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:57.495223 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.495185 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-xks-ca" pod "06c172f3-cb47-4972-b252-908b26ef5616" (UID: "06c172f3-cb47-4972-b252-908b26ef5616"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:57.538810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.527456 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.539138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.539110 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d81db8-ebfe-4ec8-80f6-b93face79b11-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.552951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.552774 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcjxt\" (UniqueName: \"kubernetes.io/projected/06c172f3-cb47-4972-b252-908b26ef5616-kube-api-access-xcjxt\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.552951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.552809 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.552951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.552825 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06c172f3-cb47-4972-b252-908b26ef5616-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.552951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.552861 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24kn5\" (UniqueName: \"kubernetes.io/projected/b5d81db8-ebfe-4ec8-80f6-b93face79b11-kube-api-access-24kn5\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:57.562417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.555164 4309 scope.go:122] "RemoveContainer" containerID="72447e44a5b22c318b7cdb72b37b8f7ce582e8f8b60a6075e2e71459f567fcc9" Nov 02 22:19:57.694574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.694544 4309 scope.go:122] "RemoveContainer" containerID="11d03f3f5291cbc74d29d186dab6a86ff55a581f1cede96a4ac3bc6e9520ef72" Nov 02 22:19:57.741401 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:19:57.741353 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29a569f1_d1b9_4866_8f40_f8a6a398e239.slice/crio-e90610af491d0bd067b1790b5bb74e9514e7e2d83cb273efdf8df10b04e08c38.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8394a652_8139_4240_b3bb_4ff8b69ea42a.slice/crio-ba56afd4a07a32460fa25e45ec27eca73d5b4abaa7e86147a713cdce1103c9e3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29a569f1_d1b9_4866_8f40_f8a6a398e239.slice/crio-conmon-e90610af491d0bd067b1790b5bb74e9514e7e2d83cb273efdf8df10b04e08c38.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8394a652_8139_4240_b3bb_4ff8b69ea42a.slice/crio-conmon-ba56afd4a07a32460fa25e45ec27eca73d5b4abaa7e86147a713cdce1103c9e3.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:19:57.806886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.806786 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" event={"ID":"b5d81db8-ebfe-4ec8-80f6-b93face79b11","Type":"ContainerDied","Data":"3094971b23b6d0b3a4accbdbcbde51a88225920f7d9ab5691c759fdf07e3ca1c"} Nov 02 22:19:57.807216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.807194 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3094971b23b6d0b3a4accbdbcbde51a88225920f7d9ab5691c759fdf07e3ca1c" Nov 02 22:19:57.807445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.807431 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd" Nov 02 22:19:57.814286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.812415 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" event={"ID":"06c172f3-cb47-4972-b252-908b26ef5616","Type":"ContainerDied","Data":"5d01e105281aa6715217e5b17ceade0541216429b25a80da68b6d9abd55111ae"} Nov 02 22:19:57.814286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.812469 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d01e105281aa6715217e5b17ceade0541216429b25a80da68b6d9abd55111ae" Nov 02 22:19:57.814286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.812555 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9" Nov 02 22:19:57.844042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.843893 4309 generic.go:356] "Generic (PLEG): container finished" podID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerID="e90610af491d0bd067b1790b5bb74e9514e7e2d83cb273efdf8df10b04e08c38" exitCode=0 Nov 02 22:19:57.844042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.843988 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" event={"ID":"29a569f1-d1b9-4866-8f40-f8a6a398e239","Type":"ContainerDied","Data":"e90610af491d0bd067b1790b5bb74e9514e7e2d83cb273efdf8df10b04e08c38"} Nov 02 22:19:57.852156 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.852125 4309 generic.go:356] "Generic (PLEG): container finished" podID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerID="ba56afd4a07a32460fa25e45ec27eca73d5b4abaa7e86147a713cdce1103c9e3" exitCode=0 Nov 02 22:19:57.852276 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.852252 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" event={"ID":"8394a652-8139-4240-b3bb-4ff8b69ea42a","Type":"ContainerDied","Data":"ba56afd4a07a32460fa25e45ec27eca73d5b4abaa7e86147a713cdce1103c9e3"} Nov 02 22:19:57.854240 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.854214 4309 generic.go:356] "Generic (PLEG): container finished" podID="13f6d309-4012-493e-a01f-c2e3acfbf6a5" containerID="e6079a91d58ed7d37a2eec9e79403ad53e16c26acbbf7c3752dc4d65aa66f6ba" exitCode=0 Nov 02 22:19:57.855279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.855252 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" event={"ID":"13f6d309-4012-493e-a01f-c2e3acfbf6a5","Type":"ContainerDied","Data":"e6079a91d58ed7d37a2eec9e79403ad53e16c26acbbf7c3752dc4d65aa66f6ba"} Nov 02 22:19:57.858604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.858580 4309 scope.go:122] "RemoveContainer" containerID="867f0012c557c399e4e5d1b46da705399efacdc410bf935c824f5d41b3cd3ea3" Nov 02 22:19:57.885370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.885331 4309 scope.go:122] "RemoveContainer" containerID="6a8f4fd64ff5de8aa02177c3cb0a1373291492e0f7ffc1fdd7f8c519f5be5dfd" Nov 02 22:19:57.920725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.920556 4309 scope.go:122] "RemoveContainer" containerID="3d372a92d354b0266c47f8370a2aa1a694ff25b08f4ae77c4a7402e7559a403c" Nov 02 22:19:57.946089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.946051 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft"] Nov 02 22:19:57.952798 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:19:57.952770 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f752b0_65ba_4b65_b25d_3506c2cf0128.slice/crio-6e9cac88582304dc6fab55c313fb3a9838039d10f37369ba51449882da95c0c9 WatchSource:0}: Error finding container 6e9cac88582304dc6fab55c313fb3a9838039d10f37369ba51449882da95c0c9: Status 404 returned error can't find the container with id 6e9cac88582304dc6fab55c313fb3a9838039d10f37369ba51449882da95c0c9 Nov 02 22:19:57.959032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.958289 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:19:57.959032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.958370 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:19:57.959032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.958410 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:19:57.996053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:57.995622 4309 scope.go:122] "RemoveContainer" containerID="0abc7e1ff1eb3430dcecfe01169bab2b74469e159160fca06ab0fa2acbb34083" Nov 02 22:19:58.082853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.082820 4309 scope.go:122] "RemoveContainer" containerID="21ecb53249a0c18bffa983502d90ab4fe4cb3afc8ecf14ecfc7199774347209e" Nov 02 22:19:58.156375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.156341 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:19:58.159350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.159331 4309 scope.go:122] "RemoveContainer" containerID="a1b6aa19aa6a0d5505110c0d70579ea3ff32fc7b8c0dd04c28cc9acba6616b51" Nov 02 22:19:58.171435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.171351 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:19:58.183942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.182469 4309 scope.go:122] "RemoveContainer" containerID="294338792e085541a0d107b39f091f69ed630200cb0ee5c1f007fb4dbcadb14e" Nov 02 22:19:58.193742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.192409 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:19:58.254013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.253883 4309 scope.go:122] "RemoveContainer" containerID="c0ea220d275d73b1d658e451ca8fa0a3fa1f4aeeb7827e251d34ed398988000f" Nov 02 22:19:58.280598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.277078 4309 scope.go:122] "RemoveContainer" containerID="9d2b4937428d2a3d1613f15162f974b2612d8270559d0813775edf8f69f3f022" Nov 02 22:19:58.291443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.291096 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-dshm\") pod \"8394a652-8139-4240-b3bb-4ff8b69ea42a\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " Nov 02 22:19:58.292022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.291759 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-home\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-home\") pod \"8394a652-8139-4240-b3bb-4ff8b69ea42a\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " Nov 02 22:19:58.292022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.291895 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a569f1-d1b9-4866-8f40-f8a6a398e239-kube-api-access-6wp7d\" (UniqueName: \"kubernetes.io/projected/29a569f1-d1b9-4866-8f40-f8a6a398e239-kube-api-access-6wp7d\") pod \"29a569f1-d1b9-4866-8f40-f8a6a398e239\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " Nov 02 22:19:58.292205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.292047 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-home" pod "8394a652-8139-4240-b3bb-4ff8b69ea42a" (UID: "8394a652-8139-4240-b3bb-4ff8b69ea42a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:58.292435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.292358 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f6d309-4012-493e-a01f-c2e3acfbf6a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f6d309-4012-493e-a01f-c2e3acfbf6a5-plugins-config-volume\") pod \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\" (UID: \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\") " Nov 02 22:19:58.292600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.292552 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-home\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-home\") pod \"29a569f1-d1b9-4866-8f40-f8a6a398e239\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " Nov 02 22:19:58.293104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.292769 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f6d309-4012-493e-a01f-c2e3acfbf6a5-plugins-config-volume" pod "13f6d309-4012-493e-a01f-c2e3acfbf6a5" (UID: "13f6d309-4012-493e-a01f-c2e3acfbf6a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:19:58.293104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.292777 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-tmp-dir\") pod \"8394a652-8139-4240-b3bb-4ff8b69ea42a\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " Nov 02 22:19:58.293104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.292927 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8394a652-8139-4240-b3bb-4ff8b69ea42a-tls-certs\" (UniqueName: \"kubernetes.io/secret/8394a652-8139-4240-b3bb-4ff8b69ea42a-tls-certs\") pod \"8394a652-8139-4240-b3bb-4ff8b69ea42a\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " Nov 02 22:19:58.293334 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.293310 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-home" pod "29a569f1-d1b9-4866-8f40-f8a6a398e239" (UID: "29a569f1-d1b9-4866-8f40-f8a6a398e239"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:58.293393 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.293364 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-tmp-dir" pod "8394a652-8139-4240-b3bb-4ff8b69ea42a" (UID: "8394a652-8139-4240-b3bb-4ff8b69ea42a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.292975 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-tmp-dir\") pod \"29a569f1-d1b9-4866-8f40-f8a6a398e239\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.293548 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-model-cache\") pod \"8394a652-8139-4240-b3bb-4ff8b69ea42a\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.293593 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f6d309-4012-493e-a01f-c2e3acfbf6a5-kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/13f6d309-4012-493e-a01f-c2e3acfbf6a5-kube-api-access-mxcm2\") pod \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\" (UID: \"13f6d309-4012-493e-a01f-c2e3acfbf6a5\") " Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.293638 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8394a652-8139-4240-b3bb-4ff8b69ea42a-kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/8394a652-8139-4240-b3bb-4ff8b69ea42a-kube-api-access-6tgvw\") pod \"8394a652-8139-4240-b3bb-4ff8b69ea42a\" (UID: \"8394a652-8139-4240-b3bb-4ff8b69ea42a\") " Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.293666 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-dshm\") pod \"29a569f1-d1b9-4866-8f40-f8a6a398e239\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.293717 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-model-cache\") pod \"29a569f1-d1b9-4866-8f40-f8a6a398e239\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.293758 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a569f1-d1b9-4866-8f40-f8a6a398e239-tls-certs\" (UniqueName: \"kubernetes.io/secret/29a569f1-d1b9-4866-8f40-f8a6a398e239-tls-certs\") pod \"29a569f1-d1b9-4866-8f40-f8a6a398e239\" (UID: \"29a569f1-d1b9-4866-8f40-f8a6a398e239\") " Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.293838 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-tmp-dir" pod "29a569f1-d1b9-4866-8f40-f8a6a398e239" (UID: "29a569f1-d1b9-4866-8f40-f8a6a398e239"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.294345 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-model-cache" pod "29a569f1-d1b9-4866-8f40-f8a6a398e239" (UID: "29a569f1-d1b9-4866-8f40-f8a6a398e239"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:58.296868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.294535 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-model-cache" pod "8394a652-8139-4240-b3bb-4ff8b69ea42a" (UID: "8394a652-8139-4240-b3bb-4ff8b69ea42a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:58.298849 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.298817 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.299006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.298979 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13f6d309-4012-493e-a01f-c2e3acfbf6a5-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.299140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.299116 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.299250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.299230 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.299364 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.299345 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.299461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.299445 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.300436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.299548 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a569f1-d1b9-4866-8f40-f8a6a398e239-kube-api-access-6wp7d" pod "29a569f1-d1b9-4866-8f40-f8a6a398e239" (UID: "29a569f1-d1b9-4866-8f40-f8a6a398e239"). InnerVolumeSpecName "kube-api-access-6wp7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:58.300436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.299558 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.300436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.299627 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-dshm" pod "8394a652-8139-4240-b3bb-4ff8b69ea42a" (UID: "8394a652-8139-4240-b3bb-4ff8b69ea42a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:58.300436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.299788 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8394a652-8139-4240-b3bb-4ff8b69ea42a-kube-api-access-6tgvw" pod "8394a652-8139-4240-b3bb-4ff8b69ea42a" (UID: "8394a652-8139-4240-b3bb-4ff8b69ea42a"). InnerVolumeSpecName "kube-api-access-6tgvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:58.301434 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.301410 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8394a652-8139-4240-b3bb-4ff8b69ea42a-tls-certs" pod "8394a652-8139-4240-b3bb-4ff8b69ea42a" (UID: "8394a652-8139-4240-b3bb-4ff8b69ea42a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:58.304174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.304126 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-dshm" pod "29a569f1-d1b9-4866-8f40-f8a6a398e239" (UID: "29a569f1-d1b9-4866-8f40-f8a6a398e239"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:19:58.307770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.307071 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a569f1-d1b9-4866-8f40-f8a6a398e239-tls-certs" pod "29a569f1-d1b9-4866-8f40-f8a6a398e239" (UID: "29a569f1-d1b9-4866-8f40-f8a6a398e239"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:19:58.307770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.307297 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f6d309-4012-493e-a01f-c2e3acfbf6a5-kube-api-access-mxcm2" pod "13f6d309-4012-493e-a01f-c2e3acfbf6a5" (UID: "13f6d309-4012-493e-a01f-c2e3acfbf6a5"). InnerVolumeSpecName "kube-api-access-mxcm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:19:58.312163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.312062 4309 scope.go:122] "RemoveContainer" containerID="2ecf5a08ef895c52b8c093ca7a943c1454852b97dd3ee592d07bc5ff057babc6" Nov 02 22:19:58.329650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.329355 4309 scope.go:122] "RemoveContainer" containerID="a2bccd24195cd9eecfdd94f12bb3c5a5a15d38c060e19dbf18990448ca8cd086" Nov 02 22:19:58.346062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.346035 4309 scope.go:122] "RemoveContainer" containerID="cbc1f5b1b9798cf0bfbaacf4891edf1c4461fa4c17279a8a5af48bd72674d734" Nov 02 22:19:58.401179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.401154 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8394a652-8139-4240-b3bb-4ff8b69ea42a-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.401385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.401181 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/13f6d309-4012-493e-a01f-c2e3acfbf6a5-kube-api-access-mxcm2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.401385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.401191 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tgvw\" (UniqueName: \"kubernetes.io/projected/8394a652-8139-4240-b3bb-4ff8b69ea42a-kube-api-access-6tgvw\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.401385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.401202 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a569f1-d1b9-4866-8f40-f8a6a398e239-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.401385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.401212 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a569f1-d1b9-4866-8f40-f8a6a398e239-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.401385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.401222 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8394a652-8139-4240-b3bb-4ff8b69ea42a-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.401385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.401235 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wp7d\" (UniqueName: \"kubernetes.io/projected/29a569f1-d1b9-4866-8f40-f8a6a398e239-kube-api-access-6wp7d\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:19:58.866001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.865966 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" event={"ID":"b4f752b0-65ba-4b65-b25d-3506c2cf0128","Type":"ContainerStarted","Data":"372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1"} Nov 02 22:19:58.866162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.866033 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" event={"ID":"b4f752b0-65ba-4b65-b25d-3506c2cf0128","Type":"ContainerStarted","Data":"6e9cac88582304dc6fab55c313fb3a9838039d10f37369ba51449882da95c0c9"} Nov 02 22:19:58.867714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.867655 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" Nov 02 22:19:58.867714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.867699 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f" event={"ID":"29a569f1-d1b9-4866-8f40-f8a6a398e239","Type":"ContainerDied","Data":"848417dce87c93a0f67e5248eeb92b44503a41b02a14be1f1a3790a6903cd82a"} Nov 02 22:19:58.867907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.867757 4309 scope.go:122] "RemoveContainer" containerID="e90610af491d0bd067b1790b5bb74e9514e7e2d83cb273efdf8df10b04e08c38" Nov 02 22:19:58.869559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.869537 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" Nov 02 22:19:58.869699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.869568 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4" event={"ID":"8394a652-8139-4240-b3bb-4ff8b69ea42a","Type":"ContainerDied","Data":"300c741f92ba229575f8be5c30d15bf8464a29d8bdac9e565e890c193093100b"} Nov 02 22:19:58.871492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.871470 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" event={"ID":"13f6d309-4012-493e-a01f-c2e3acfbf6a5","Type":"ContainerDied","Data":"545982258ff2898a3f12062e9d00e6b91f4b7893c5da8a2d300fb4d192b9c7a1"} Nov 02 22:19:58.871608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.871581 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6" Nov 02 22:19:58.880995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.880962 4309 scope.go:122] "RemoveContainer" containerID="ba56afd4a07a32460fa25e45ec27eca73d5b4abaa7e86147a713cdce1103c9e3" Nov 02 22:19:58.897810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.897576 4309 scope.go:122] "RemoveContainer" containerID="e6079a91d58ed7d37a2eec9e79403ad53e16c26acbbf7c3752dc4d65aa66f6ba" Nov 02 22:19:58.919824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.919765 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" podStartSLOduration=5.919745546 podStartE2EDuration="5.919745546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:19:58.903472531 +0000 UTC m=+1326.841585750" watchObservedRunningTime="2026-11-02 22:19:58.919745546 +0000 UTC m=+1326.857858765" Nov 02 22:19:58.962073 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.962043 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f"] Nov 02 22:19:58.971216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.971179 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/distinct-w6-a1d78c-kserve-7dd8865bfc-58b8f"] Nov 02 22:19:58.980271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.980243 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4"] Nov 02 22:19:58.988629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.988598 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a1d78c/simulator-w6-a1d78c-kserve-6b7bfc7567-tl9v4"] Nov 02 22:19:58.996131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:58.996100 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6"] Nov 02 22:19:59.000388 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:59.000363 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5f4a2f-8459d9bf67-8f8g6"] Nov 02 22:19:59.172042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:59.172008 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz"] Nov 02 22:19:59.176658 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:59.176634 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-5f4a2f-b8skz"] Nov 02 22:19:59.612545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:59.612517 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:59.617970 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:59.617940 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:59.887309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:59.887201 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:19:59.888475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:19:59.888448 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:20:00.191650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:00.191521 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f6d309-4012-493e-a01f-c2e3acfbf6a5" path="/var/lib/kubelet/pods/13f6d309-4012-493e-a01f-c2e3acfbf6a5/volumes" Nov 02 22:20:00.192303 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:00.192283 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" path="/var/lib/kubelet/pods/29a569f1-d1b9-4866-8f40-f8a6a398e239/volumes" Nov 02 22:20:00.192884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:00.192860 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356ea8f5-7786-44c5-90c3-064b8dd2c626" path="/var/lib/kubelet/pods/356ea8f5-7786-44c5-90c3-064b8dd2c626/volumes" Nov 02 22:20:00.193663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:00.193646 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" path="/var/lib/kubelet/pods/8394a652-8139-4240-b3bb-4ff8b69ea42a/volumes" Nov 02 22:20:00.234177 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:00.234144 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5"] Nov 02 22:20:00.234356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:00.234336 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" containerID="cri-o://fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc" gracePeriod=30 Nov 02 22:20:02.388575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.388538 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w"] Nov 02 22:20:02.390380 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390347 4309 cpu_manager.go:414] "containerMap: removing container" podUID="06c172f3-cb47-4972-b252-908b26ef5616" containerName="revoke-keys" Nov 02 22:20:02.390496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390381 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c172f3-cb47-4972-b252-908b26ef5616" containerName="revoke-keys" Nov 02 22:20:02.390496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390438 4309 cpu_manager.go:414] "containerMap: removing container" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" Nov 02 22:20:02.390496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390446 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" Nov 02 22:20:02.390614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390496 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" Nov 02 22:20:02.390614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390504 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" Nov 02 22:20:02.390614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390538 4309 cpu_manager.go:414] "containerMap: removing container" podUID="13f6d309-4012-493e-a01f-c2e3acfbf6a5" containerName="payload-pre-processing" Nov 02 22:20:02.390614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390545 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f6d309-4012-493e-a01f-c2e3acfbf6a5" containerName="payload-pre-processing" Nov 02 22:20:02.390614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390562 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b5d81db8-ebfe-4ec8-80f6-b93face79b11" containerName="revoke-keys" Nov 02 22:20:02.390614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390570 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d81db8-ebfe-4ec8-80f6-b93face79b11" containerName="revoke-keys" Nov 02 22:20:02.390960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390943 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8394a652-8139-4240-b3bb-4ff8b69ea42a" containerName="main" Nov 02 22:20:02.391111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.390984 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f6d309-4012-493e-a01f-c2e3acfbf6a5" containerName="payload-pre-processing" Nov 02 22:20:02.391111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.391057 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d81db8-ebfe-4ec8-80f6-b93face79b11" containerName="revoke-keys" Nov 02 22:20:02.391111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.391072 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c172f3-cb47-4972-b252-908b26ef5616" containerName="revoke-keys" Nov 02 22:20:02.400262 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.400230 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a569f1-d1b9-4866-8f40-f8a6a398e239" containerName="main" Nov 02 22:20:02.411901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.411856 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w"] Nov 02 22:20:02.412209 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.412186 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:02.422721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.421471 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a54217\"" Nov 02 22:20:02.422721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.422039 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a54217-dockercfg-f5pcw\"" Nov 02 22:20:02.441853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.441822 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w\" (UID: \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:02.442017 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.441975 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mq9g\" (UniqueName: \"kubernetes.io/projected/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-kube-api-access-4mq9g\") pod \"payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w\" (UID: \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:02.445160 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.444921 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn"] Nov 02 22:20:02.452404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.452111 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:02.464189 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.464153 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn"] Nov 02 22:20:02.544125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.544075 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w\" (UID: \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:02.544337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.544161 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqz5r\" (UniqueName: \"kubernetes.io/projected/e7a03376-19f0-4899-ad29-66280272e7d6-kube-api-access-rqz5r\") pod \"payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn\" (UID: \"e7a03376-19f0-4899-ad29-66280272e7d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:02.544337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.544244 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mq9g\" (UniqueName: \"kubernetes.io/projected/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-kube-api-access-4mq9g\") pod \"payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w\" (UID: \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:02.544438 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.544361 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a03376-19f0-4899-ad29-66280272e7d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn\" (UID: \"e7a03376-19f0-4899-ad29-66280272e7d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:02.545237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.545211 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w\" (UID: \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:02.553115 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.553082 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mq9g\" (UniqueName: \"kubernetes.io/projected/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-kube-api-access-4mq9g\") pod \"payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w\" (UID: \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:02.645746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.645642 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a03376-19f0-4899-ad29-66280272e7d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn\" (UID: \"e7a03376-19f0-4899-ad29-66280272e7d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:02.646045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.646015 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqz5r\" (UniqueName: \"kubernetes.io/projected/e7a03376-19f0-4899-ad29-66280272e7d6-kube-api-access-rqz5r\") pod \"payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn\" (UID: \"e7a03376-19f0-4899-ad29-66280272e7d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:02.646577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.646556 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a03376-19f0-4899-ad29-66280272e7d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn\" (UID: \"e7a03376-19f0-4899-ad29-66280272e7d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:02.655577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.655543 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqz5r\" (UniqueName: \"kubernetes.io/projected/e7a03376-19f0-4899-ad29-66280272e7d6-kube-api-access-rqz5r\") pod \"payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn\" (UID: \"e7a03376-19f0-4899-ad29-66280272e7d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:02.745698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.745000 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:02.786698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:02.776370 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:03.027049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.026979 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5"] Nov 02 22:20:03.037555 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.037506 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1c75b42-vszf5"] Nov 02 22:20:03.270825 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:20:03.270795 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc1bd6d_fcec_4658_9968_4bd9bd401d15.slice/crio-dcfd7e92fad19ecc7b5b9906fba87d20d3225efc117d4106232512b1f25181ec WatchSource:0}: Error finding container dcfd7e92fad19ecc7b5b9906fba87d20d3225efc117d4106232512b1f25181ec: Status 404 returned error can't find the container with id dcfd7e92fad19ecc7b5b9906fba87d20d3225efc117d4106232512b1f25181ec Nov 02 22:20:03.274992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.274970 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w"] Nov 02 22:20:03.282052 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.282020 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn"] Nov 02 22:20:03.980891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.980850 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" event={"ID":"e7a03376-19f0-4899-ad29-66280272e7d6","Type":"ContainerStarted","Data":"3283cc8a8fe5792f4a2270c2222b4eb03d371b31401e5332529283193ec22b95"} Nov 02 22:20:03.981458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.981432 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" event={"ID":"e7a03376-19f0-4899-ad29-66280272e7d6","Type":"ContainerStarted","Data":"4a49a18cfb67026984c479868cd05b962cdaec6cc7afe27309e76c78b41938ef"} Nov 02 22:20:03.982545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.982524 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:03.985302 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.985278 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" event={"ID":"ffc1bd6d-fcec-4658-9968-4bd9bd401d15","Type":"ContainerStarted","Data":"7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d"} Nov 02 22:20:03.985439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.985326 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" event={"ID":"ffc1bd6d-fcec-4658-9968-4bd9bd401d15","Type":"ContainerStarted","Data":"dcfd7e92fad19ecc7b5b9906fba87d20d3225efc117d4106232512b1f25181ec"} Nov 02 22:20:03.985564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:03.985489 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:04.052739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:04.046606 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" podStartSLOduration=2.046587164 podStartE2EDuration="2.046587164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:04.044957142 +0000 UTC m=+1331.983070361" watchObservedRunningTime="2026-11-02 22:20:04.046587164 +0000 UTC m=+1331.984700383" Nov 02 22:20:04.052739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:04.051118 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" podStartSLOduration=2.051104543 podStartE2EDuration="2.051104543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:04.01600407 +0000 UTC m=+1331.954117286" watchObservedRunningTime="2026-11-02 22:20:04.051104543 +0000 UTC m=+1331.989217774" Nov 02 22:20:04.198887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:04.198849 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af95b18e-259a-4658-b937-f26fe81fbd20" path="/var/lib/kubelet/pods/af95b18e-259a-4658-b937-f26fe81fbd20/volumes" Nov 02 22:20:05.543632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.543600 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:20:05.598494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.598459 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-envoy\") pod \"5e9e0e9b-0636-4f92-9251-171ea770c528\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.598515 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-kube-api-access-484r7\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-kube-api-access-484r7\") pod \"5e9e0e9b-0636-4f92-9251-171ea770c528\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.598584 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-token\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-token\") pod \"5e9e0e9b-0636-4f92-9251-171ea770c528\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.598745 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-socket\") pod \"5e9e0e9b-0636-4f92-9251-171ea770c528\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.598801 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-podinfo\") pod \"5e9e0e9b-0636-4f92-9251-171ea770c528\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.598915 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-credential-socket\") pod \"5e9e0e9b-0636-4f92-9251-171ea770c528\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.598967 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-data\") pod \"5e9e0e9b-0636-4f92-9251-171ea770c528\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.599078 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9e0e9b-0636-4f92-9251-171ea770c528-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9e0e9b-0636-4f92-9251-171ea770c528-istiod-ca-cert\") pod \"5e9e0e9b-0636-4f92-9251-171ea770c528\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.599112 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-certs\") pod \"5e9e0e9b-0636-4f92-9251-171ea770c528\" (UID: \"5e9e0e9b-0636-4f92-9251-171ea770c528\") " Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.600477 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-certs" pod "5e9e0e9b-0636-4f92-9251-171ea770c528" (UID: "5e9e0e9b-0636-4f92-9251-171ea770c528"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.601731 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-credential-socket" pod "5e9e0e9b-0636-4f92-9251-171ea770c528" (UID: "5e9e0e9b-0636-4f92-9251-171ea770c528"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.601804 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-socket" pod "5e9e0e9b-0636-4f92-9251-171ea770c528" (UID: "5e9e0e9b-0636-4f92-9251-171ea770c528"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.601945 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9e0e9b-0636-4f92-9251-171ea770c528-istiod-ca-cert" pod "5e9e0e9b-0636-4f92-9251-171ea770c528" (UID: "5e9e0e9b-0636-4f92-9251-171ea770c528"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.601993 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-data" pod "5e9e0e9b-0636-4f92-9251-171ea770c528" (UID: "5e9e0e9b-0636-4f92-9251-171ea770c528"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.605992 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-envoy" pod "5e9e0e9b-0636-4f92-9251-171ea770c528" (UID: "5e9e0e9b-0636-4f92-9251-171ea770c528"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:05.612695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.611104 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-podinfo" pod "5e9e0e9b-0636-4f92-9251-171ea770c528" (UID: "5e9e0e9b-0636-4f92-9251-171ea770c528"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:05.619230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.619208 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-kube-api-access-484r7" pod "5e9e0e9b-0636-4f92-9251-171ea770c528" (UID: "5e9e0e9b-0636-4f92-9251-171ea770c528"). InnerVolumeSpecName "kube-api-access-484r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:05.621206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.621168 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-token" pod "5e9e0e9b-0636-4f92-9251-171ea770c528" (UID: "5e9e0e9b-0636-4f92-9251-171ea770c528"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:05.702021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.701978 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e9e0e9b-0636-4f92-9251-171ea770c528-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:05.702021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.702022 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:05.702021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.702038 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:05.702021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.702052 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-484r7\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-kube-api-access-484r7\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:05.702307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.702068 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:05.702307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.702081 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:05.702307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.702094 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:05.702307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.702107 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:05.702307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:05.702121 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e9e0e9b-0636-4f92-9251-171ea770c528-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:06.017684 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.017636 4309 generic.go:356] "Generic (PLEG): container finished" podID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerID="fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc" exitCode=0 Nov 02 22:20:06.017873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.017714 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" event={"ID":"5e9e0e9b-0636-4f92-9251-171ea770c528","Type":"ContainerDied","Data":"fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc"} Nov 02 22:20:06.017873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.017735 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" Nov 02 22:20:06.017873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.017821 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5" event={"ID":"5e9e0e9b-0636-4f92-9251-171ea770c528","Type":"ContainerDied","Data":"9a5483f2ec3f78078dafbcdd488e5126c7291fb32aa7ffb8f27988ace26b50d2"} Nov 02 22:20:06.017873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.017854 4309 scope.go:122] "RemoveContainer" containerID="fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc" Nov 02 22:20:06.032695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.032642 4309 scope.go:122] "RemoveContainer" containerID="fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc" Nov 02 22:20:06.033086 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:06.033058 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc\": container with ID starting with fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc not found: ID does not exist" containerID="fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc" Nov 02 22:20:06.033178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.033099 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc"} err="failed to get container status \"fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc\": rpc error: code = NotFound desc = could not find container \"fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc\": container with ID starting with fde1183a048ec6f981fdf4ddcd66ef4d3c36fa93026183f76bfff3293b7cf8dc not found: ID does not exist" Nov 02 22:20:06.046007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.045977 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5"] Nov 02 22:20:06.050630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.050606 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-6wfg5"] Nov 02 22:20:06.194574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.194535 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" path="/var/lib/kubelet/pods/5e9e0e9b-0636-4f92-9251-171ea770c528/volumes" Nov 02 22:20:06.609908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.605823 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z"] Nov 02 22:20:06.609908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.607145 4309 cpu_manager.go:414] "containerMap: removing container" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" Nov 02 22:20:06.609908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.607167 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" Nov 02 22:20:06.609908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.607363 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9e0e9b-0636-4f92-9251-171ea770c528" containerName="istio-proxy" Nov 02 22:20:06.623131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.623101 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z"] Nov 02 22:20:06.623303 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.623242 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.628940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.628915 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e49edd-openshift-default-dockercfg-qk94g\"" Nov 02 22:20:06.730801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.729305 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.730801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.729455 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.730801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.729502 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.730801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.729592 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.730801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.729633 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq479\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-kube-api-access-dq479\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.730801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.729775 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.730801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.729879 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.730801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.729943 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.730801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.729984 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.761376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.761339 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh"] Nov 02 22:20:06.761753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.761720 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" containerID="cri-o://559f582933eaad7c2a09f4ab89d8caeab3e662637e27cf7f6fb7f4664a2576d7" gracePeriod=30 Nov 02 22:20:06.766805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.766410 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd"] Nov 02 22:20:06.766805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.766715 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" containerID="cri-o://9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee" gracePeriod=30 Nov 02 22:20:06.786490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.786452 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv"] Nov 02 22:20:06.786724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.786694 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" containerID="cri-o://a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2" gracePeriod=30 Nov 02 22:20:06.876749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.869004 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq479\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-kube-api-access-dq479\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.876749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.869209 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.876749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.869315 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.876749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.869358 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.876749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.869395 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.876749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.869468 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.876749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.869608 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.894932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.890730 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.894932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.890907 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.894932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.891508 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.894932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.892518 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.894932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.893159 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.894932 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.893405 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:06.908168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:06.907136 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:07.093697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.092976 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq479\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-kube-api-access-dq479\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:07.093697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.093250 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:07.115698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.112530 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:07.149750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.149330 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:07.249311 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.249277 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:07.577201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.577171 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z"] Nov 02 22:20:07.577773 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:20:07.577740 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b47633d_d531_4e0c_a9ab_2942027ae6cd.slice/crio-3b66deb99eee4740219acd334531a417734a15bcdaf04078418a90d8bbb69c8f WatchSource:0}: Error finding container 3b66deb99eee4740219acd334531a417734a15bcdaf04078418a90d8bbb69c8f: Status 404 returned error can't find the container with id 3b66deb99eee4740219acd334531a417734a15bcdaf04078418a90d8bbb69c8f Nov 02 22:20:07.581976 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.581936 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:20:07.582072 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.582024 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:20:07.582147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.582130 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:20:07.659447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.659386 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:20:07.661463 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:07.661413 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:20:08.393704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.380459 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c"] Nov 02 22:20:08.423788 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.420408 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.470702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.454875 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-credential-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.470702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.455011 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82133f4f-73f3-42a8-8bb9-02673f70fe28-istiod-ca-cert\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.470702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.455059 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-podinfo\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.470702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.455105 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-certs\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.470702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.455138 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-data\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.470702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.455250 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xtl\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-kube-api-access-f2xtl\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.470702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.455509 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-envoy\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.470702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.455564 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.470702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.455703 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-token\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.480377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.479592 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c"] Nov 02 22:20:08.523521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.517884 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" event={"ID":"4b47633d-d531-4e0c-a9ab-2942027ae6cd","Type":"ContainerStarted","Data":"4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d"} Nov 02 22:20:08.524448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.523781 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" event={"ID":"4b47633d-d531-4e0c-a9ab-2942027ae6cd","Type":"ContainerStarted","Data":"3b66deb99eee4740219acd334531a417734a15bcdaf04078418a90d8bbb69c8f"} Nov 02 22:20:08.550057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.544447 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-7bb56e-openshift-default-dockercfg-jkclr\"" Nov 02 22:20:08.569996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.569958 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-envoy\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.611858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.611817 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.612253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.612224 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-token\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.612522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.612498 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.612606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.612504 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-credential-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.640890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.612696 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82133f4f-73f3-42a8-8bb9-02673f70fe28-istiod-ca-cert\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.640890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.612752 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-podinfo\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.640890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.612860 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-certs\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.640890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.612886 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-data\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.640890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.612994 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xtl\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-kube-api-access-f2xtl\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.640890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.614376 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82133f4f-73f3-42a8-8bb9-02673f70fe28-istiod-ca-cert\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.672769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.643877 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-credential-socket\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.702018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.644294 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-data\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.702226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.644514 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-certs\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.725723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.703933 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podStartSLOduration=2.703912125 podStartE2EDuration="2.703912125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:08.570634531 +0000 UTC m=+1336.508747750" watchObservedRunningTime="2026-11-02 22:20:08.703912125 +0000 UTC m=+1336.642025344" Nov 02 22:20:08.795815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.791716 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-token\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.797554 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.797026 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-podinfo\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.798251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.798213 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-envoy\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:08.842520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:08.842364 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xtl\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-kube-api-access-f2xtl\") pod \"e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:09.050508 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.050429 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:09.250205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.250177 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:09.255263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.255237 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:09.358470 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:20:09.358430 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82133f4f_73f3_42a8_8bb9_02673f70fe28.slice/crio-764dc7255925d04e047e5dca30160ed82822bea6ec45abd61b8191fcf2bf1c98 WatchSource:0}: Error finding container 764dc7255925d04e047e5dca30160ed82822bea6ec45abd61b8191fcf2bf1c98: Status 404 returned error can't find the container with id 764dc7255925d04e047e5dca30160ed82822bea6ec45abd61b8191fcf2bf1c98 Nov 02 22:20:09.359549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.359526 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c"] Nov 02 22:20:09.361352 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.361325 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:20:09.361407 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.361384 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:20:09.361449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.361411 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:20:09.599813 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.599770 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" event={"ID":"82133f4f-73f3-42a8-8bb9-02673f70fe28","Type":"ContainerStarted","Data":"fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f"} Nov 02 22:20:09.600070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.600056 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:09.600212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.600197 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" event={"ID":"82133f4f-73f3-42a8-8bb9-02673f70fe28","Type":"ContainerStarted","Data":"764dc7255925d04e047e5dca30160ed82822bea6ec45abd61b8191fcf2bf1c98"} Nov 02 22:20:09.610751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.604982 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:20:09.629135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:09.629077 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" podStartSLOduration=1.6290633319999999 podStartE2EDuration="1.629063332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:09.626438548 +0000 UTC m=+1337.564551769" watchObservedRunningTime="2026-11-02 22:20:09.629063332 +0000 UTC m=+1337.567176551" Nov 02 22:20:10.051225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:10.051133 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:10.052314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:10.052287 4309 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 02 22:20:10.052433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:10.052334 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 02 22:20:11.058098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:11.058059 4309 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 02 22:20:11.058597 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:11.058129 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 02 22:20:11.343758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:11.337578 4309 patch_prober.go:29] interesting pod/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:20:11.345157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:11.345122 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:20:11.763432 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:11.763324 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:20:11.951126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:11.951054 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw"] Nov 02 22:20:11.956950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:11.956924 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a1d78c-fb2bw"] Nov 02 22:20:12.052699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.051826 4309 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 02 22:20:12.052699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.051904 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 02 22:20:12.253844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.253802 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fce0df8-64fb-428f-938c-b542d53ec8d8" path="/var/lib/kubelet/pods/0fce0df8-64fb-428f-938c-b542d53ec8d8/volumes" Nov 02 22:20:12.505723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.494865 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:20:12.550591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.550562 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8"] Nov 02 22:20:12.552094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.552071 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" Nov 02 22:20:12.552426 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.552408 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" Nov 02 22:20:12.552835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.552818 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8282cc88-823a-429b-9385-478d41cd1bea" containerName="istio-proxy" Nov 02 22:20:12.558901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.558884 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:12.560012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.559987 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w"] Nov 02 22:20:12.561225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.561201 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:20:12.561528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.561501 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" Nov 02 22:20:12.564134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.564096 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e49edd\"" Nov 02 22:20:12.566109 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.566088 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.580624 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-certs\") pod \"8282cc88-823a-429b-9385-478d41cd1bea\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.580719 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-envoy\") pod \"8282cc88-823a-429b-9385-478d41cd1bea\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.580754 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-data\") pod \"8282cc88-823a-429b-9385-478d41cd1bea\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.580927 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-socket\") pod \"8282cc88-823a-429b-9385-478d41cd1bea\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.580974 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-istio-token\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-istio-token\") pod \"8282cc88-823a-429b-9385-478d41cd1bea\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.581008 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8282cc88-823a-429b-9385-478d41cd1bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8282cc88-823a-429b-9385-478d41cd1bea-istiod-ca-cert\") pod \"8282cc88-823a-429b-9385-478d41cd1bea\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.581188 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8282cc88-823a-429b-9385-478d41cd1bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8282cc88-823a-429b-9385-478d41cd1bea-istio-podinfo\") pod \"8282cc88-823a-429b-9385-478d41cd1bea\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.581278 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-kube-api-access-9trvc\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-kube-api-access-9trvc\") pod \"8282cc88-823a-429b-9385-478d41cd1bea\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.581386 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-credential-socket\") pod \"8282cc88-823a-429b-9385-478d41cd1bea\" (UID: \"8282cc88-823a-429b-9385-478d41cd1bea\") " Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.597666 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8"] Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.599946 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-certs" pod "8282cc88-823a-429b-9385-478d41cd1bea" (UID: "8282cc88-823a-429b-9385-478d41cd1bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:12.603059 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.600274 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8282cc88-823a-429b-9385-478d41cd1bea-istiod-ca-cert" pod "8282cc88-823a-429b-9385-478d41cd1bea" (UID: "8282cc88-823a-429b-9385-478d41cd1bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:12.669705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.608400 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e49edd-dockercfg-g2h2g\"" Nov 02 22:20:12.669705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.650352 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-credential-socket" pod "8282cc88-823a-429b-9385-478d41cd1bea" (UID: "8282cc88-823a-429b-9385-478d41cd1bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:12.955004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.861254 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-token\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-token\") pod \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " Nov 02 22:20:12.955004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.861396 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-socket\") pod \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " Nov 02 22:20:12.955004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.861479 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-credential-socket\") pod \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " Nov 02 22:20:12.955004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.861841 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-certs\") pod \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " Nov 02 22:20:12.955004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.861900 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-kube-api-access-wx4l6\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-kube-api-access-wx4l6\") pod \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " Nov 02 22:20:12.955004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.862011 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-data\") pod \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " Nov 02 22:20:12.955004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.862350 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-envoy\") pod \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " Nov 02 22:20:12.955004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.862600 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istiod-ca-cert\") pod \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " Nov 02 22:20:12.955004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:12.862695 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-podinfo\") pod \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\" (UID: \"95a6102b-8f0b-4ecc-8e29-836c57f0420b\") " Nov 02 22:20:13.069738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.060353 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-istio-token" pod "8282cc88-823a-429b-9385-478d41cd1bea" (UID: "8282cc88-823a-429b-9385-478d41cd1bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:13.069738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.060521 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-envoy" pod "8282cc88-823a-429b-9385-478d41cd1bea" (UID: "8282cc88-823a-429b-9385-478d41cd1bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.074745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.073204 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8282cc88-823a-429b-9385-478d41cd1bea-istio-podinfo" pod "8282cc88-823a-429b-9385-478d41cd1bea" (UID: "8282cc88-823a-429b-9385-478d41cd1bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:13.074745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.074254 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7csb\" (UniqueName: \"kubernetes.io/projected/8f07b490-666a-4652-9c73-b7021c90a5e7-kube-api-access-c7csb\") pod \"payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w\" (UID: \"8f07b490-666a-4652-9c73-b7021c90a5e7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:13.074745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.074362 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f07b490-666a-4652-9c73-b7021c90a5e7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w\" (UID: \"8f07b490-666a-4652-9c73-b7021c90a5e7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:13.074745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.074488 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwf57\" (UniqueName: \"kubernetes.io/projected/38542064-c103-49d1-9ce6-f0fd1398ac97-kube-api-access-cwf57\") pod \"payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8\" (UID: \"38542064-c103-49d1-9ce6-f0fd1398ac97\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:13.075578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.075542 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38542064-c103-49d1-9ce6-f0fd1398ac97-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8\" (UID: \"38542064-c103-49d1-9ce6-f0fd1398ac97\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:13.077732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.077711 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8282cc88-823a-429b-9385-478d41cd1bea-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.077960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.077943 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.078110 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.078090 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.078254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.078226 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.078362 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.078349 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.078485 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.078470 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8282cc88-823a-429b-9385-478d41cd1bea-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.079385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.079338 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-kube-api-access-9trvc" pod "8282cc88-823a-429b-9385-478d41cd1bea" (UID: "8282cc88-823a-429b-9385-478d41cd1bea"). InnerVolumeSpecName "kube-api-access-9trvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:13.081005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.080089 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-data" pod "8282cc88-823a-429b-9385-478d41cd1bea" (UID: "8282cc88-823a-429b-9385-478d41cd1bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.081005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.080176 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-socket" pod "8282cc88-823a-429b-9385-478d41cd1bea" (UID: "8282cc88-823a-429b-9385-478d41cd1bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.081451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.081246 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-socket" pod "95a6102b-8f0b-4ecc-8e29-836c57f0420b" (UID: "95a6102b-8f0b-4ecc-8e29-836c57f0420b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.089176 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.081607 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-credential-socket" pod "95a6102b-8f0b-4ecc-8e29-836c57f0420b" (UID: "95a6102b-8f0b-4ecc-8e29-836c57f0420b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.089748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.089350 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-certs" pod "95a6102b-8f0b-4ecc-8e29-836c57f0420b" (UID: "95a6102b-8f0b-4ecc-8e29-836c57f0420b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.121330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.115358 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-kube-api-access-wx4l6" pod "95a6102b-8f0b-4ecc-8e29-836c57f0420b" (UID: "95a6102b-8f0b-4ecc-8e29-836c57f0420b"). InnerVolumeSpecName "kube-api-access-wx4l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:13.132127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.121591 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-data" pod "95a6102b-8f0b-4ecc-8e29-836c57f0420b" (UID: "95a6102b-8f0b-4ecc-8e29-836c57f0420b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.132560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.132422 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istiod-ca-cert" pod "95a6102b-8f0b-4ecc-8e29-836c57f0420b" (UID: "95a6102b-8f0b-4ecc-8e29-836c57f0420b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:13.141409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.137056 4309 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 02 22:20:13.141409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.137119 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 02 22:20:13.151315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.151284 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w"] Nov 02 22:20:13.154960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.154927 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-envoy" pod "95a6102b-8f0b-4ecc-8e29-836c57f0420b" (UID: "95a6102b-8f0b-4ecc-8e29-836c57f0420b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.172347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.172302 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-token" pod "95a6102b-8f0b-4ecc-8e29-836c57f0420b" (UID: "95a6102b-8f0b-4ecc-8e29-836c57f0420b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:13.177113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.177084 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-podinfo" pod "95a6102b-8f0b-4ecc-8e29-836c57f0420b" (UID: "95a6102b-8f0b-4ecc-8e29-836c57f0420b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:13.180577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.180548 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38542064-c103-49d1-9ce6-f0fd1398ac97-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8\" (UID: \"38542064-c103-49d1-9ce6-f0fd1398ac97\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:13.181350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.180875 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7csb\" (UniqueName: \"kubernetes.io/projected/8f07b490-666a-4652-9c73-b7021c90a5e7-kube-api-access-c7csb\") pod \"payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w\" (UID: \"8f07b490-666a-4652-9c73-b7021c90a5e7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:13.181350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.180943 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f07b490-666a-4652-9c73-b7021c90a5e7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w\" (UID: \"8f07b490-666a-4652-9c73-b7021c90a5e7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:13.181350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.181023 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwf57\" (UniqueName: \"kubernetes.io/projected/38542064-c103-49d1-9ce6-f0fd1398ac97-kube-api-access-cwf57\") pod \"payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8\" (UID: \"38542064-c103-49d1-9ce6-f0fd1398ac97\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:13.181591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.181355 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38542064-c103-49d1-9ce6-f0fd1398ac97-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8\" (UID: \"38542064-c103-49d1-9ce6-f0fd1398ac97\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:13.181591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.181400 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182355 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f07b490-666a-4652-9c73-b7021c90a5e7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w\" (UID: \"8f07b490-666a-4652-9c73-b7021c90a5e7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182404 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182423 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trvc\" (UniqueName: \"kubernetes.io/projected/8282cc88-823a-429b-9385-478d41cd1bea-kube-api-access-9trvc\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182441 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182455 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182470 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182484 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182498 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8282cc88-823a-429b-9385-478d41cd1bea-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182512 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182526 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182539 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95a6102b-8f0b-4ecc-8e29-836c57f0420b-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.190717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.182554 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx4l6\" (UniqueName: \"kubernetes.io/projected/95a6102b-8f0b-4ecc-8e29-836c57f0420b-kube-api-access-wx4l6\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.225608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.222754 4309 generic.go:356] "Generic (PLEG): container finished" podID="8282cc88-823a-429b-9385-478d41cd1bea" containerID="a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2" exitCode=0 Nov 02 22:20:13.225608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.222921 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" event={"ID":"8282cc88-823a-429b-9385-478d41cd1bea","Type":"ContainerDied","Data":"a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2"} Nov 02 22:20:13.225608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.222964 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" event={"ID":"8282cc88-823a-429b-9385-478d41cd1bea","Type":"ContainerDied","Data":"041f622e9bde244cb64f1b99e3673508cd01b38172279df1f4a06661af379859"} Nov 02 22:20:13.225608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.222984 4309 scope.go:122] "RemoveContainer" containerID="a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2" Nov 02 22:20:13.225608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.223123 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv" Nov 02 22:20:13.225608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.225199 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwf57\" (UniqueName: \"kubernetes.io/projected/38542064-c103-49d1-9ce6-f0fd1398ac97-kube-api-access-cwf57\") pod \"payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8\" (UID: \"38542064-c103-49d1-9ce6-f0fd1398ac97\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:13.228891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.228859 4309 generic.go:356] "Generic (PLEG): container finished" podID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerID="9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee" exitCode=0 Nov 02 22:20:13.229026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.228958 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" event={"ID":"95a6102b-8f0b-4ecc-8e29-836c57f0420b","Type":"ContainerDied","Data":"9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee"} Nov 02 22:20:13.229597 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.229009 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" event={"ID":"95a6102b-8f0b-4ecc-8e29-836c57f0420b","Type":"ContainerDied","Data":"30e6177e4696ceeed8ef6c195334bd187d89fd8766bf98644b907ad7ed5da3e0"} Nov 02 22:20:13.229750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.229732 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd" Nov 02 22:20:13.238455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.238356 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7csb\" (UniqueName: \"kubernetes.io/projected/8f07b490-666a-4652-9c73-b7021c90a5e7-kube-api-access-c7csb\") pod \"payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w\" (UID: \"8f07b490-666a-4652-9c73-b7021c90a5e7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:13.243050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.243027 4309 generic.go:356] "Generic (PLEG): container finished" podID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerID="559f582933eaad7c2a09f4ab89d8caeab3e662637e27cf7f6fb7f4664a2576d7" exitCode=0 Nov 02 22:20:13.243318 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.243297 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" event={"ID":"72bd580c-fb5e-448f-a476-ceb46734b65b","Type":"ContainerDied","Data":"559f582933eaad7c2a09f4ab89d8caeab3e662637e27cf7f6fb7f4664a2576d7"} Nov 02 22:20:13.262245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.262216 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv"] Nov 02 22:20:13.274778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.274743 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-h7vcv"] Nov 02 22:20:13.274913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.274898 4309 scope.go:122] "RemoveContainer" containerID="a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2" Nov 02 22:20:13.276339 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:13.276225 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2\": container with ID starting with a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2 not found: ID does not exist" containerID="a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2" Nov 02 22:20:13.276339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.276270 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2"} err="failed to get container status \"a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2\": rpc error: code = NotFound desc = could not find container \"a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2\": container with ID starting with a81752ac62bf979cc21cd4a77bced5f6067506027cc7a32a96f90939577bb3e2 not found: ID does not exist" Nov 02 22:20:13.276339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.276289 4309 scope.go:122] "RemoveContainer" containerID="9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee" Nov 02 22:20:13.334385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.333564 4309 scope.go:122] "RemoveContainer" containerID="afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e" Nov 02 22:20:13.337914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.337890 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd"] Nov 02 22:20:13.344414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.344030 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-ftmvd"] Nov 02 22:20:13.349790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.349707 4309 scope.go:122] "RemoveContainer" containerID="9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee" Nov 02 22:20:13.350147 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:13.350041 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee\": container with ID starting with 9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee not found: ID does not exist" containerID="9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee" Nov 02 22:20:13.350147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.350068 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee"} err="failed to get container status \"9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee\": rpc error: code = NotFound desc = could not find container \"9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee\": container with ID starting with 9452c05add28e93943abc6d5112eee24a6e784a60ea157f13a471472e34091ee not found: ID does not exist" Nov 02 22:20:13.350147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.350087 4309 scope.go:122] "RemoveContainer" containerID="afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e" Nov 02 22:20:13.350357 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:13.350313 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e\": container with ID starting with afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e not found: ID does not exist" containerID="afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e" Nov 02 22:20:13.350357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.350336 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e"} err="failed to get container status \"afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e\": rpc error: code = NotFound desc = could not find container \"afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e\": container with ID starting with afd2aed8f7ac34524749a07c3a131fa78533e7dd5fe4f1eb044717c60990e34e not found: ID does not exist" Nov 02 22:20:13.373139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.373118 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:20:13.381071 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.381052 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438369 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-credential-socket\") pod \"72bd580c-fb5e-448f-a476-ceb46734b65b\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438524 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72bd580c-fb5e-448f-a476-ceb46734b65b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bd580c-fb5e-448f-a476-ceb46734b65b-istiod-ca-cert\") pod \"72bd580c-fb5e-448f-a476-ceb46734b65b\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438558 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-socket\") pod \"72bd580c-fb5e-448f-a476-ceb46734b65b\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438630 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-token\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-token\") pod \"72bd580c-fb5e-448f-a476-ceb46734b65b\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438652 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-credential-socket" pod "72bd580c-fb5e-448f-a476-ceb46734b65b" (UID: "72bd580c-fb5e-448f-a476-ceb46734b65b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438695 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-podinfo\") pod \"72bd580c-fb5e-448f-a476-ceb46734b65b\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438725 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-envoy\") pod \"72bd580c-fb5e-448f-a476-ceb46734b65b\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438764 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-kube-api-access-l5h4b\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-kube-api-access-l5h4b\") pod \"72bd580c-fb5e-448f-a476-ceb46734b65b\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438818 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-data\") pod \"72bd580c-fb5e-448f-a476-ceb46734b65b\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438917 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-certs\") pod \"72bd580c-fb5e-448f-a476-ceb46734b65b\" (UID: \"72bd580c-fb5e-448f-a476-ceb46734b65b\") " Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.438952 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-socket" pod "72bd580c-fb5e-448f-a476-ceb46734b65b" (UID: "72bd580c-fb5e-448f-a476-ceb46734b65b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.439572 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bd580c-fb5e-448f-a476-ceb46734b65b-istiod-ca-cert" pod "72bd580c-fb5e-448f-a476-ceb46734b65b" (UID: "72bd580c-fb5e-448f-a476-ceb46734b65b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:13.440006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.439856 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-certs" pod "72bd580c-fb5e-448f-a476-ceb46734b65b" (UID: "72bd580c-fb5e-448f-a476-ceb46734b65b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.440910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.440399 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-data" pod "72bd580c-fb5e-448f-a476-ceb46734b65b" (UID: "72bd580c-fb5e-448f-a476-ceb46734b65b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.440910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.440761 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.440910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.440786 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bd580c-fb5e-448f-a476-ceb46734b65b-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.440910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.440801 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.442632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.442595 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-token" pod "72bd580c-fb5e-448f-a476-ceb46734b65b" (UID: "72bd580c-fb5e-448f-a476-ceb46734b65b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:13.443829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.443807 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-podinfo" pod "72bd580c-fb5e-448f-a476-ceb46734b65b" (UID: "72bd580c-fb5e-448f-a476-ceb46734b65b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:13.443943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.443816 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-envoy" pod "72bd580c-fb5e-448f-a476-ceb46734b65b" (UID: "72bd580c-fb5e-448f-a476-ceb46734b65b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:13.445592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.445573 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-kube-api-access-l5h4b" pod "72bd580c-fb5e-448f-a476-ceb46734b65b" (UID: "72bd580c-fb5e-448f-a476-ceb46734b65b"). InnerVolumeSpecName "kube-api-access-l5h4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:13.472767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.472725 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:13.552075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.551937 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.552075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.551963 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.552075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.551973 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.552075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.551983 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5h4b\" (UniqueName: \"kubernetes.io/projected/72bd580c-fb5e-448f-a476-ceb46734b65b-kube-api-access-l5h4b\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.552075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.551993 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.552075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.552001 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bd580c-fb5e-448f-a476-ceb46734b65b-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:13.772091 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:13.772056 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8"] Nov 02 22:20:14.037454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.037430 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w"] Nov 02 22:20:14.051618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.051594 4309 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 02 22:20:14.051779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.051639 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 02 22:20:14.228252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.228216 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8282cc88-823a-429b-9385-478d41cd1bea" path="/var/lib/kubelet/pods/8282cc88-823a-429b-9385-478d41cd1bea/volumes" Nov 02 22:20:14.229024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.229007 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" path="/var/lib/kubelet/pods/95a6102b-8f0b-4ecc-8e29-836c57f0420b/volumes" Nov 02 22:20:14.254740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.254668 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" event={"ID":"8f07b490-666a-4652-9c73-b7021c90a5e7","Type":"ContainerStarted","Data":"1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5"} Nov 02 22:20:14.254902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.254767 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" event={"ID":"8f07b490-666a-4652-9c73-b7021c90a5e7","Type":"ContainerStarted","Data":"794f41940466e31e94b2bf3df822d9c4f9a2955f1c44ef4aab6bf0e3128ae991"} Nov 02 22:20:14.257127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.257102 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" Nov 02 22:20:14.257241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.257099 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh" event={"ID":"72bd580c-fb5e-448f-a476-ceb46734b65b","Type":"ContainerDied","Data":"f0937ff2d2e1aed3c25287c841a4e55e3a740f5e5021eb50a84240d5062992fd"} Nov 02 22:20:14.257241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.257235 4309 scope.go:122] "RemoveContainer" containerID="559f582933eaad7c2a09f4ab89d8caeab3e662637e27cf7f6fb7f4664a2576d7" Nov 02 22:20:14.258950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.258927 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" event={"ID":"38542064-c103-49d1-9ce6-f0fd1398ac97","Type":"ContainerStarted","Data":"0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa"} Nov 02 22:20:14.259090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.258971 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" event={"ID":"38542064-c103-49d1-9ce6-f0fd1398ac97","Type":"ContainerStarted","Data":"077bb4e9e7ef48a0da9bd3b2725eb9cde7eada889e5c6be9d0752da947aa1d40"} Nov 02 22:20:14.259090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.258992 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:14.281184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.281138 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" podStartSLOduration=2.281124057 podStartE2EDuration="2.281124057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:14.27704951 +0000 UTC m=+1342.215162728" watchObservedRunningTime="2026-11-02 22:20:14.281124057 +0000 UTC m=+1342.219237275" Nov 02 22:20:14.325433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.325370 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" podStartSLOduration=2.325350335 podStartE2EDuration="2.325350335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:14.311050566 +0000 UTC m=+1342.249163808" watchObservedRunningTime="2026-11-02 22:20:14.325350335 +0000 UTC m=+1342.263463554" Nov 02 22:20:14.343825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.343792 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh"] Nov 02 22:20:14.350826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:14.350798 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5f4a2f-openshift-default-9d5fc57c5-rdpbh"] Nov 02 22:20:15.015541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:15.015508 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:20:15.051504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:15.051469 4309 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 02 22:20:15.051690 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:15.051522 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 02 22:20:15.274074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:15.273999 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:15.509026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:15.493828 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85"] Nov 02 22:20:15.520027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:15.519987 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" containerID="cri-o://6931dbf3e31ced6b76061acbd6dd269bb50f77847e50eefa4b0d8db81f926ba6" gracePeriod=30 Nov 02 22:20:16.021243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:16.021213 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:20:16.088186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:16.088149 4309 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 02 22:20:16.088474 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:16.088442 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 02 22:20:16.276155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:16.276061 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" path="/var/lib/kubelet/pods/72bd580c-fb5e-448f-a476-ceb46734b65b/volumes" Nov 02 22:20:17.056631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:17.056596 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:17.056631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:17.057893 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:17.056631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:17.059001 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:20:20.771697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:20.768613 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp"] Nov 02 22:20:20.771697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:20.769809 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" containerID="cri-o://d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93" gracePeriod=30 Nov 02 22:20:20.778755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:20.777571 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx"] Nov 02 22:20:20.778755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:20.777844 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" containerID="cri-o://1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3" gracePeriod=30 Nov 02 22:20:20.784538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:20.784454 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw"] Nov 02 22:20:20.787086 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:20.784762 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" containerID="cri-o://6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380" gracePeriod=30 Nov 02 22:20:21.433752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:21.427441 4309 generic.go:356] "Generic (PLEG): container finished" podID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerID="6931dbf3e31ced6b76061acbd6dd269bb50f77847e50eefa4b0d8db81f926ba6" exitCode=0 Nov 02 22:20:21.433752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:21.427821 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" event={"ID":"52b21749-629f-416a-9ad1-5ab65ec9ccd8","Type":"ContainerDied","Data":"6931dbf3e31ced6b76061acbd6dd269bb50f77847e50eefa4b0d8db81f926ba6"} Nov 02 22:20:22.223842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.215847 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:20:22.351235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351188 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-socket\") pod \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " Nov 02 22:20:22.351431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351269 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-podinfo\") pod \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " Nov 02 22:20:22.351431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351346 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-credential-socket\") pod \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " Nov 02 22:20:22.351431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351411 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-envoy\") pod \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " Nov 02 22:20:22.351610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351458 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-data\") pod \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " Nov 02 22:20:22.351610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351502 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-socket" pod "52b21749-629f-416a-9ad1-5ab65ec9ccd8" (UID: "52b21749-629f-416a-9ad1-5ab65ec9ccd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:22.351610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351557 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-kube-api-access-vf4l2\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-kube-api-access-vf4l2\") pod \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " Nov 02 22:20:22.351777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351645 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-token\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-token\") pod \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " Nov 02 22:20:22.351777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351689 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istiod-ca-cert\") pod \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " Nov 02 22:20:22.351899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351804 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-credential-socket" pod "52b21749-629f-416a-9ad1-5ab65ec9ccd8" (UID: "52b21749-629f-416a-9ad1-5ab65ec9ccd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:22.351957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351898 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-data" pod "52b21749-629f-416a-9ad1-5ab65ec9ccd8" (UID: "52b21749-629f-416a-9ad1-5ab65ec9ccd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:22.351957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.351944 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-certs\") pod \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\" (UID: \"52b21749-629f-416a-9ad1-5ab65ec9ccd8\") " Nov 02 22:20:22.352146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.352122 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istiod-ca-cert" pod "52b21749-629f-416a-9ad1-5ab65ec9ccd8" (UID: "52b21749-629f-416a-9ad1-5ab65ec9ccd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:22.357080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.353038 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.357080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.353062 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.357080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.353078 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.357080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.353091 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.357080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.355627 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-certs" pod "52b21749-629f-416a-9ad1-5ab65ec9ccd8" (UID: "52b21749-629f-416a-9ad1-5ab65ec9ccd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:22.357080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.355831 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-envoy" pod "52b21749-629f-416a-9ad1-5ab65ec9ccd8" (UID: "52b21749-629f-416a-9ad1-5ab65ec9ccd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:22.357080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.355871 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-token" pod "52b21749-629f-416a-9ad1-5ab65ec9ccd8" (UID: "52b21749-629f-416a-9ad1-5ab65ec9ccd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:22.357664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.357640 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-podinfo" pod "52b21749-629f-416a-9ad1-5ab65ec9ccd8" (UID: "52b21749-629f-416a-9ad1-5ab65ec9ccd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:22.357842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.357819 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-kube-api-access-vf4l2" pod "52b21749-629f-416a-9ad1-5ab65ec9ccd8" (UID: "52b21749-629f-416a-9ad1-5ab65ec9ccd8"). InnerVolumeSpecName "kube-api-access-vf4l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:22.459650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.459534 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf4l2\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-kube-api-access-vf4l2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.459650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.459574 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.459650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.459593 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.459650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.459609 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.459650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.459622 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52b21749-629f-416a-9ad1-5ab65ec9ccd8-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:22.517698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.511359 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" event={"ID":"52b21749-629f-416a-9ad1-5ab65ec9ccd8","Type":"ContainerDied","Data":"d231a13cdb47b615f700b8c7424beff06f7a3fe41deb2ac1fb3d8f10bbb5a7dc"} Nov 02 22:20:22.517698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.511446 4309 scope.go:122] "RemoveContainer" containerID="6931dbf3e31ced6b76061acbd6dd269bb50f77847e50eefa4b0d8db81f926ba6" Nov 02 22:20:22.517698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.511629 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85" Nov 02 22:20:22.718746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.715723 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85"] Nov 02 22:20:22.731667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:22.730021 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-89p85"] Nov 02 22:20:24.192980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:24.192938 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" path="/var/lib/kubelet/pods/52b21749-629f-416a-9ad1-5ab65ec9ccd8/volumes" Nov 02 22:20:24.339730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:24.339671 4309 patch_prober.go:29] interesting pod/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:20:24.339875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:24.339745 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:20:25.277054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:25.277027 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:20:26.271746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.271417 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:20:26.278376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.277282 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:20:26.335158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.335011 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:20:26.375144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.375120 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:20:26.435320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.435293 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-kube-api-access-7b8qn\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-kube-api-access-7b8qn\") pod \"e7ff4254-f98d-427f-be0a-440c773b870c\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " Nov 02 22:20:26.435589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.435575 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-socket\") pod \"e7ff4254-f98d-427f-be0a-440c773b870c\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " Nov 02 22:20:26.435747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.435735 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-certs\") pod \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " Nov 02 22:20:26.435875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.435866 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-envoy\") pod \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " Nov 02 22:20:26.435958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.435950 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-credential-socket\") pod \"e7ff4254-f98d-427f-be0a-440c773b870c\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " Nov 02 22:20:26.436143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.436114 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-certs\") pod \"e7ff4254-f98d-427f-be0a-440c773b870c\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " Nov 02 22:20:26.436616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.436312 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-istio-token\") pod \"e7ff4254-f98d-427f-be0a-440c773b870c\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " Nov 02 22:20:26.437103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.436533 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-certs" pod "e7ff4254-f98d-427f-be0a-440c773b870c" (UID: "e7ff4254-f98d-427f-be0a-440c773b870c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.437103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437066 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-credential-socket" pod "e7ff4254-f98d-427f-be0a-440c773b870c" (UID: "e7ff4254-f98d-427f-be0a-440c773b870c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.437103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.436960 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-certs" pod "908dd6a3-83b4-439d-a9e1-e76b8d8648e0" (UID: "908dd6a3-83b4-439d-a9e1-e76b8d8648e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.437103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437066 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ff4254-f98d-427f-be0a-440c773b870c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff4254-f98d-427f-be0a-440c773b870c-istiod-ca-cert\") pod \"e7ff4254-f98d-427f-be0a-440c773b870c\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " Nov 02 22:20:26.437386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437119 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-socket" pod "e7ff4254-f98d-427f-be0a-440c773b870c" (UID: "e7ff4254-f98d-427f-be0a-440c773b870c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.437386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437165 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-credential-socket\") pod \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " Nov 02 22:20:26.437386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437203 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-socket\") pod \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " Nov 02 22:20:26.437577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437458 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-credential-socket" pod "908dd6a3-83b4-439d-a9e1-e76b8d8648e0" (UID: "908dd6a3-83b4-439d-a9e1-e76b8d8648e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.437577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437381 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ff4254-f98d-427f-be0a-440c773b870c-istiod-ca-cert" pod "e7ff4254-f98d-427f-be0a-440c773b870c" (UID: "e7ff4254-f98d-427f-be0a-440c773b870c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:26.437577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437562 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-socket" pod "908dd6a3-83b4-439d-a9e1-e76b8d8648e0" (UID: "908dd6a3-83b4-439d-a9e1-e76b8d8648e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.437776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437627 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-data\") pod \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " Nov 02 22:20:26.437776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437734 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7ff4254-f98d-427f-be0a-440c773b870c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff4254-f98d-427f-be0a-440c773b870c-istio-podinfo\") pod \"e7ff4254-f98d-427f-be0a-440c773b870c\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " Nov 02 22:20:26.437776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437766 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-kube-api-access-pkxg5\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-kube-api-access-pkxg5\") pod \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " Nov 02 22:20:26.437969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437818 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-podinfo\") pod \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " Nov 02 22:20:26.437969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437853 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-data\") pod \"e7ff4254-f98d-427f-be0a-440c773b870c\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " Nov 02 22:20:26.437969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437916 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-token\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-token\") pod \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " Nov 02 22:20:26.437969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.437957 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-envoy\") pod \"e7ff4254-f98d-427f-be0a-440c773b870c\" (UID: \"e7ff4254-f98d-427f-be0a-440c773b870c\") " Nov 02 22:20:26.438196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.438010 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istiod-ca-cert\") pod \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\" (UID: \"908dd6a3-83b4-439d-a9e1-e76b8d8648e0\") " Nov 02 22:20:26.438752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.438637 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-data" pod "e7ff4254-f98d-427f-be0a-440c773b870c" (UID: "e7ff4254-f98d-427f-be0a-440c773b870c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.439348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.439067 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.439348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.439095 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff4254-f98d-427f-be0a-440c773b870c-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.439348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.439111 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.439348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.439127 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.439348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.439143 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.439348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.439159 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.439348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.439173 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.439348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.439186 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.440145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.439386 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istiod-ca-cert" pod "908dd6a3-83b4-439d-a9e1-e76b8d8648e0" (UID: "908dd6a3-83b4-439d-a9e1-e76b8d8648e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:26.440409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.440385 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-data" pod "908dd6a3-83b4-439d-a9e1-e76b8d8648e0" (UID: "908dd6a3-83b4-439d-a9e1-e76b8d8648e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.447223 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.447198 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-podinfo" pod "908dd6a3-83b4-439d-a9e1-e76b8d8648e0" (UID: "908dd6a3-83b4-439d-a9e1-e76b8d8648e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:26.447322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.447219 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-token" pod "908dd6a3-83b4-439d-a9e1-e76b8d8648e0" (UID: "908dd6a3-83b4-439d-a9e1-e76b8d8648e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:26.447389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.447357 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-kube-api-access-7b8qn" pod "e7ff4254-f98d-427f-be0a-440c773b870c" (UID: "e7ff4254-f98d-427f-be0a-440c773b870c"). InnerVolumeSpecName "kube-api-access-7b8qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:26.447440 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.447394 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-istio-token" pod "e7ff4254-f98d-427f-be0a-440c773b870c" (UID: "e7ff4254-f98d-427f-be0a-440c773b870c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:26.447545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.447527 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7ff4254-f98d-427f-be0a-440c773b870c-istio-podinfo" pod "e7ff4254-f98d-427f-be0a-440c773b870c" (UID: "e7ff4254-f98d-427f-be0a-440c773b870c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:26.447656 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.447636 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-envoy" pod "e7ff4254-f98d-427f-be0a-440c773b870c" (UID: "e7ff4254-f98d-427f-be0a-440c773b870c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.447999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.447976 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-envoy" pod "908dd6a3-83b4-439d-a9e1-e76b8d8648e0" (UID: "908dd6a3-83b4-439d-a9e1-e76b8d8648e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.449238 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.449202 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-kube-api-access-pkxg5" pod "908dd6a3-83b4-439d-a9e1-e76b8d8648e0" (UID: "908dd6a3-83b4-439d-a9e1-e76b8d8648e0"). InnerVolumeSpecName "kube-api-access-pkxg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:26.540323 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540294 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-istio-token\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-istio-token\") pod \"639a7482-b746-42ea-a91e-95b7a39cd269\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " Nov 02 22:20:26.540551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540331 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639a7482-b746-42ea-a91e-95b7a39cd269-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639a7482-b746-42ea-a91e-95b7a39cd269-istiod-ca-cert\") pod \"639a7482-b746-42ea-a91e-95b7a39cd269\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " Nov 02 22:20:26.540551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540466 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-kube-api-access-swvvb\") pod \"639a7482-b746-42ea-a91e-95b7a39cd269\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " Nov 02 22:20:26.540551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540522 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-envoy\") pod \"639a7482-b746-42ea-a91e-95b7a39cd269\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " Nov 02 22:20:26.540823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540567 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-certs\") pod \"639a7482-b746-42ea-a91e-95b7a39cd269\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " Nov 02 22:20:26.540823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540657 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-socket\") pod \"639a7482-b746-42ea-a91e-95b7a39cd269\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " Nov 02 22:20:26.540823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540750 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/639a7482-b746-42ea-a91e-95b7a39cd269-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639a7482-b746-42ea-a91e-95b7a39cd269-istio-podinfo\") pod \"639a7482-b746-42ea-a91e-95b7a39cd269\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " Nov 02 22:20:26.540823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540818 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-credential-socket\") pod \"639a7482-b746-42ea-a91e-95b7a39cd269\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " Nov 02 22:20:26.541031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540939 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-data\") pod \"639a7482-b746-42ea-a91e-95b7a39cd269\" (UID: \"639a7482-b746-42ea-a91e-95b7a39cd269\") " Nov 02 22:20:26.541206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540936 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-certs" pod "639a7482-b746-42ea-a91e-95b7a39cd269" (UID: "639a7482-b746-42ea-a91e-95b7a39cd269"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.541206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.540948 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639a7482-b746-42ea-a91e-95b7a39cd269-istiod-ca-cert" pod "639a7482-b746-42ea-a91e-95b7a39cd269" (UID: "639a7482-b746-42ea-a91e-95b7a39cd269"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:20:26.541374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541145 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-data" pod "639a7482-b746-42ea-a91e-95b7a39cd269" (UID: "639a7482-b746-42ea-a91e-95b7a39cd269"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.541374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541163 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-credential-socket" pod "639a7482-b746-42ea-a91e-95b7a39cd269" (UID: "639a7482-b746-42ea-a91e-95b7a39cd269"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.541374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541173 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-socket" pod "639a7482-b746-42ea-a91e-95b7a39cd269" (UID: "639a7482-b746-42ea-a91e-95b7a39cd269"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.541820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541799 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/639a7482-b746-42ea-a91e-95b7a39cd269-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541819 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541828 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541837 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541846 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541855 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff4254-f98d-427f-be0a-440c773b870c-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541865 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkxg5\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-kube-api-access-pkxg5\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541875 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541885 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541894 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541904 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff4254-f98d-427f-be0a-440c773b870c-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541912 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/908dd6a3-83b4-439d-a9e1-e76b8d8648e0-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541920 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541928 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b8qn\" (UniqueName: \"kubernetes.io/projected/e7ff4254-f98d-427f-be0a-440c773b870c-kube-api-access-7b8qn\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.541993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.541936 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.542742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.542654 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-istio-token" pod "639a7482-b746-42ea-a91e-95b7a39cd269" (UID: "639a7482-b746-42ea-a91e-95b7a39cd269"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:26.543152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.543132 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-kube-api-access-swvvb" pod "639a7482-b746-42ea-a91e-95b7a39cd269" (UID: "639a7482-b746-42ea-a91e-95b7a39cd269"). InnerVolumeSpecName "kube-api-access-swvvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:20:26.543272 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.543240 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/639a7482-b746-42ea-a91e-95b7a39cd269-istio-podinfo" pod "639a7482-b746-42ea-a91e-95b7a39cd269" (UID: "639a7482-b746-42ea-a91e-95b7a39cd269"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:20:26.543272 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.543256 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-envoy" pod "639a7482-b746-42ea-a91e-95b7a39cd269" (UID: "639a7482-b746-42ea-a91e-95b7a39cd269"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:20:26.552431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.552405 4309 generic.go:356] "Generic (PLEG): container finished" podID="639a7482-b746-42ea-a91e-95b7a39cd269" containerID="6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380" exitCode=0 Nov 02 22:20:26.552526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.552453 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" Nov 02 22:20:26.552526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.552486 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" event={"ID":"639a7482-b746-42ea-a91e-95b7a39cd269","Type":"ContainerDied","Data":"6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380"} Nov 02 22:20:26.552623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.552540 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw" event={"ID":"639a7482-b746-42ea-a91e-95b7a39cd269","Type":"ContainerDied","Data":"8bd2a61e2724e9bf07c9c6022b17301eb80fa5853d266845e3afdffbad87db4f"} Nov 02 22:20:26.552623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.552574 4309 scope.go:122] "RemoveContainer" containerID="6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380" Nov 02 22:20:26.554187 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.554160 4309 generic.go:356] "Generic (PLEG): container finished" podID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerID="1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3" exitCode=0 Nov 02 22:20:26.554295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.554209 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" Nov 02 22:20:26.554295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.554251 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" event={"ID":"908dd6a3-83b4-439d-a9e1-e76b8d8648e0","Type":"ContainerDied","Data":"1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3"} Nov 02 22:20:26.554443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.554298 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx" event={"ID":"908dd6a3-83b4-439d-a9e1-e76b8d8648e0","Type":"ContainerDied","Data":"b6e1c9d5db4699f3570aafcfd623aa39f54e254fd8a5dc132b1a67101f23ec4d"} Nov 02 22:20:26.555953 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.555932 4309 generic.go:356] "Generic (PLEG): container finished" podID="e7ff4254-f98d-427f-be0a-440c773b870c" containerID="d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93" exitCode=0 Nov 02 22:20:26.556060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.555995 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" Nov 02 22:20:26.556148 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.556051 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" event={"ID":"e7ff4254-f98d-427f-be0a-440c773b870c","Type":"ContainerDied","Data":"d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93"} Nov 02 22:20:26.556148 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.556128 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp" event={"ID":"e7ff4254-f98d-427f-be0a-440c773b870c","Type":"ContainerDied","Data":"f87a17dae955ebb03bbeb14f35b89a891105d3f479b6d49aa14540a916cb95b3"} Nov 02 22:20:26.564791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.564771 4309 scope.go:122] "RemoveContainer" containerID="9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d" Nov 02 22:20:26.581395 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.581374 4309 scope.go:122] "RemoveContainer" containerID="6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380" Nov 02 22:20:26.581792 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:26.581766 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380\": container with ID starting with 6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380 not found: ID does not exist" containerID="6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380" Nov 02 22:20:26.581923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.581801 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380"} err="failed to get container status \"6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380\": rpc error: code = NotFound desc = could not find container \"6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380\": container with ID starting with 6a3e67dc3c30f44327e6624aa602f5200bb655b913168ca6d6a6548a5cc7b380 not found: ID does not exist" Nov 02 22:20:26.581923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.581827 4309 scope.go:122] "RemoveContainer" containerID="9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d" Nov 02 22:20:26.582157 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:26.582133 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d\": container with ID starting with 9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d not found: ID does not exist" containerID="9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d" Nov 02 22:20:26.582218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.582166 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d"} err="failed to get container status \"9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d\": rpc error: code = NotFound desc = could not find container \"9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d\": container with ID starting with 9039ba04d24b62907d864e8c36d339f27d35f63f23453562e8e6c15a6442e20d not found: ID does not exist" Nov 02 22:20:26.582218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.582191 4309 scope.go:122] "RemoveContainer" containerID="1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3" Nov 02 22:20:26.588638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.588603 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw"] Nov 02 22:20:26.593247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.593223 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-9zkpw"] Nov 02 22:20:26.595703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.595684 4309 scope.go:122] "RemoveContainer" containerID="1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3" Nov 02 22:20:26.595977 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:26.595952 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3\": container with ID starting with 1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3 not found: ID does not exist" containerID="1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3" Nov 02 22:20:26.596108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.596086 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3"} err="failed to get container status \"1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3\": rpc error: code = NotFound desc = could not find container \"1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3\": container with ID starting with 1247f943987a1bd036325fbefed349256819784a3fc2bee4caec15ea25e03bf3 not found: ID does not exist" Nov 02 22:20:26.596108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.596108 4309 scope.go:122] "RemoveContainer" containerID="d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93" Nov 02 22:20:26.623137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.623068 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx"] Nov 02 22:20:26.633324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.633302 4309 scope.go:122] "RemoveContainer" containerID="d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93" Nov 02 22:20:26.633722 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:26.633697 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93\": container with ID starting with d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93 not found: ID does not exist" containerID="d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93" Nov 02 22:20:26.633809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.633731 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93"} err="failed to get container status \"d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93\": rpc error: code = NotFound desc = could not find container \"d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93\": container with ID starting with d7066fde52dae82430d02f586faf3899e773c4df426ebbed2d6a84e9f5239b93 not found: ID does not exist" Nov 02 22:20:26.634987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.634963 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-l46dx"] Nov 02 22:20:26.639111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.639088 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp"] Nov 02 22:20:26.642681 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.642656 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a1d78c-openshift-default-69b647f777-cjrwp"] Nov 02 22:20:26.643592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.643574 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.643638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.643597 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvvb\" (UniqueName: \"kubernetes.io/projected/639a7482-b746-42ea-a91e-95b7a39cd269-kube-api-access-swvvb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.643638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.643607 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/639a7482-b746-42ea-a91e-95b7a39cd269-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:26.643638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:26.643617 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/639a7482-b746-42ea-a91e-95b7a39cd269-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:20:28.192147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:28.192115 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" path="/var/lib/kubelet/pods/639a7482-b746-42ea-a91e-95b7a39cd269/volumes" Nov 02 22:20:28.192896 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:28.192878 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" path="/var/lib/kubelet/pods/908dd6a3-83b4-439d-a9e1-e76b8d8648e0/volumes" Nov 02 22:20:28.193540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:28.193515 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" path="/var/lib/kubelet/pods/e7ff4254-f98d-427f-be0a-440c773b870c/volumes" Nov 02 22:20:35.046263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.046231 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2"] Nov 02 22:20:35.047042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047023 4309 cpu_manager.go:414] "containerMap: removing container" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" Nov 02 22:20:35.047042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047042 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047056 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047061 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047075 4309 cpu_manager.go:414] "containerMap: removing container" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047080 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047094 4309 cpu_manager.go:414] "containerMap: removing container" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047100 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047126 4309 cpu_manager.go:414] "containerMap: removing container" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047130 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047140 4309 cpu_manager.go:414] "containerMap: removing container" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047146 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047154 4309 cpu_manager.go:414] "containerMap: removing container" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" Nov 02 22:20:35.047164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047158 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" Nov 02 22:20:35.047524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047173 4309 cpu_manager.go:414] "containerMap: removing container" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" Nov 02 22:20:35.047524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047177 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" Nov 02 22:20:35.047524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047278 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" Nov 02 22:20:35.047524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047286 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bd580c-fb5e-448f-a476-ceb46734b65b" containerName="istio-proxy" Nov 02 22:20:35.047524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047296 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b21749-629f-416a-9ad1-5ab65ec9ccd8" containerName="istio-proxy" Nov 02 22:20:35.047524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047304 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a6102b-8f0b-4ecc-8e29-836c57f0420b" containerName="istio-proxy" Nov 02 22:20:35.047524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047314 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ff4254-f98d-427f-be0a-440c773b870c" containerName="istio-proxy" Nov 02 22:20:35.047524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.047322 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="908dd6a3-83b4-439d-a9e1-e76b8d8648e0" containerName="istio-proxy" Nov 02 22:20:35.053021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.052999 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:35.060344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.060316 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d28bd1\"" Nov 02 22:20:35.061024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.061003 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2"] Nov 02 22:20:35.061548 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.061522 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d28bd1-dockercfg-vqb6s\"" Nov 02 22:20:35.150410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.150372 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357afe48-05c2-4529-b51f-196c8aacc233-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2\" (UID: \"357afe48-05c2-4529-b51f-196c8aacc233\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:35.150670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.150425 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmvv\" (UniqueName: \"kubernetes.io/projected/357afe48-05c2-4529-b51f-196c8aacc233-kube-api-access-jmmvv\") pod \"payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2\" (UID: \"357afe48-05c2-4529-b51f-196c8aacc233\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:35.256992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.256955 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357afe48-05c2-4529-b51f-196c8aacc233-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2\" (UID: \"357afe48-05c2-4529-b51f-196c8aacc233\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:35.271300 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.271269 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmvv\" (UniqueName: \"kubernetes.io/projected/357afe48-05c2-4529-b51f-196c8aacc233-kube-api-access-jmmvv\") pod \"payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2\" (UID: \"357afe48-05c2-4529-b51f-196c8aacc233\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:35.271992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.269615 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357afe48-05c2-4529-b51f-196c8aacc233-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2\" (UID: \"357afe48-05c2-4529-b51f-196c8aacc233\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:35.303741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.303615 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmvv\" (UniqueName: \"kubernetes.io/projected/357afe48-05c2-4529-b51f-196c8aacc233-kube-api-access-jmmvv\") pod \"payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2\" (UID: \"357afe48-05c2-4529-b51f-196c8aacc233\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:35.393973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.393717 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:35.786653 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:20:35.786623 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357afe48_05c2_4529_b51f_196c8aacc233.slice/crio-9720420658f94929c2f0acdeee8cc0017b534482a0341c9864894959375e0365 WatchSource:0}: Error finding container 9720420658f94929c2f0acdeee8cc0017b534482a0341c9864894959375e0365: Status 404 returned error can't find the container with id 9720420658f94929c2f0acdeee8cc0017b534482a0341c9864894959375e0365 Nov 02 22:20:35.788033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:35.787968 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2"] Nov 02 22:20:36.804697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:36.801958 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" event={"ID":"357afe48-05c2-4529-b51f-196c8aacc233","Type":"ContainerStarted","Data":"48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac"} Nov 02 22:20:36.804697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:36.802029 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" event={"ID":"357afe48-05c2-4529-b51f-196c8aacc233","Type":"ContainerStarted","Data":"9720420658f94929c2f0acdeee8cc0017b534482a0341c9864894959375e0365"} Nov 02 22:20:36.804697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:36.803270 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:36.841689 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:36.841628 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" podStartSLOduration=1.841614691 podStartE2EDuration="1.841614691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:36.838609651 +0000 UTC m=+1364.776722870" watchObservedRunningTime="2026-11-02 22:20:36.841614691 +0000 UTC m=+1364.779727909" Nov 02 22:20:41.598921 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:20:41.598869 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:20:42.077014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.076983 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj"] Nov 02 22:20:42.077972 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.077952 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="639a7482-b746-42ea-a91e-95b7a39cd269" containerName="istio-proxy" Nov 02 22:20:42.083055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.083038 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:42.087868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.087842 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a621f1-dockercfg-mvwrf\"" Nov 02 22:20:42.087993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.087960 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a621f1\"" Nov 02 22:20:42.090726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.090701 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj"] Nov 02 22:20:42.299139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.299097 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lpnr\" (UniqueName: \"kubernetes.io/projected/3a6a2ece-b673-4597-8a52-2b9bb819f81d-kube-api-access-5lpnr\") pod \"payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj\" (UID: \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:42.309695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.301782 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a6a2ece-b673-4597-8a52-2b9bb819f81d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj\" (UID: \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:42.416744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.415840 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lpnr\" (UniqueName: \"kubernetes.io/projected/3a6a2ece-b673-4597-8a52-2b9bb819f81d-kube-api-access-5lpnr\") pod \"payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj\" (UID: \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:42.416744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.415939 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a6a2ece-b673-4597-8a52-2b9bb819f81d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj\" (UID: \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:42.431712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.417237 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a6a2ece-b673-4597-8a52-2b9bb819f81d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj\" (UID: \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:42.462481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.457925 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lpnr\" (UniqueName: \"kubernetes.io/projected/3a6a2ece-b673-4597-8a52-2b9bb819f81d-kube-api-access-5lpnr\") pod \"payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj\" (UID: \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:42.717846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:42.717763 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:43.010448 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:20:43.010418 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6a2ece_b673_4597_8a52_2b9bb819f81d.slice/crio-81bf15e1413d36f37f4e2e959dd48bc6cd6961b51253ac453b297f29df665ec6 WatchSource:0}: Error finding container 81bf15e1413d36f37f4e2e959dd48bc6cd6961b51253ac453b297f29df665ec6: Status 404 returned error can't find the container with id 81bf15e1413d36f37f4e2e959dd48bc6cd6961b51253ac453b297f29df665ec6 Nov 02 22:20:43.011145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:43.011096 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj"] Nov 02 22:20:43.950191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:43.950153 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" event={"ID":"3a6a2ece-b673-4597-8a52-2b9bb819f81d","Type":"ContainerStarted","Data":"66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379"} Nov 02 22:20:43.950703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:43.950208 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" event={"ID":"3a6a2ece-b673-4597-8a52-2b9bb819f81d","Type":"ContainerStarted","Data":"81bf15e1413d36f37f4e2e959dd48bc6cd6961b51253ac453b297f29df665ec6"} Nov 02 22:20:43.950703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:43.950240 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:43.981726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:43.975122 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" podStartSLOduration=1.9751030969999999 podStartE2EDuration="1.975103097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:20:43.966912351 +0000 UTC m=+1371.905025570" watchObservedRunningTime="2026-11-02 22:20:43.975103097 +0000 UTC m=+1371.913216319" Nov 02 22:20:48.924706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:48.923879 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:20:54.964807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:54.964777 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:20:58.571010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:58.570975 4309 scope.go:122] "RemoveContainer" containerID="ff04574973b59e5ee210410ad769c2b7eda9b62e879c4b2b856fa1334297a94d" Nov 02 22:20:58.662465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:58.661090 4309 scope.go:122] "RemoveContainer" containerID="c49131e60c4d863e6023011fda02e324a8137247a19f9aa59fbaf66776a91313" Nov 02 22:20:58.703912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:58.703879 4309 scope.go:122] "RemoveContainer" containerID="974b811ceffb2156bbb0db411d641cadff72f65ed1e1f2631fe3cc3d18f472eb" Nov 02 22:20:58.772802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:58.772771 4309 scope.go:122] "RemoveContainer" containerID="d9f4717b45e9bdecc417f8fd8b5ad7425621ac342483367d52d67c62922ffd7c" Nov 02 22:20:58.843147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:58.843120 4309 scope.go:122] "RemoveContainer" containerID="3721229251e4619ac669e4197d4a81324c0dfaeddce64663439155329fb18584" Nov 02 22:20:59.090710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:59.089878 4309 scope.go:122] "RemoveContainer" containerID="aeed214d63fa011f853df5a1e337906a41ac2566ab10e2fc40b7b6e831f72c0f" Nov 02 22:20:59.165705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:20:59.165029 4309 scope.go:122] "RemoveContainer" containerID="1ef47db951c0e6978b7a877b27f371be263ae3f64ad2e5ad4625977d4d8c7c26" Nov 02 22:21:04.829304 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.829269 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz"] Nov 02 22:21:04.840732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.837610 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:04.852735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.845021 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5eba37-openshift-default-dockercfg-6psk2\"" Nov 02 22:21:04.856799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.853692 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz"] Nov 02 22:21:04.939708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.934034 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-credential-socket\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:04.939708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.934084 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hvz7\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-kube-api-access-4hvz7\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:04.939708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.934155 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-socket\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:04.939708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.934245 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-podinfo\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:04.939708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.934272 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-envoy\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:04.939708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.934322 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-certs\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:04.939708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.934398 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-token\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:04.939708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.934443 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istiod-ca-cert\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:04.939708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:04.934496 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-data\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.036156 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-credential-socket\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.036201 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hvz7\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-kube-api-access-4hvz7\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.036270 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-socket\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.036356 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-podinfo\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.036381 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-envoy\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.036424 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-certs\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.036520 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-token\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.036572 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istiod-ca-cert\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.036623 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-data\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.037320 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-data\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.037590 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-credential-socket\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.038191 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-socket\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.038916 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-certs\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.224870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.041118 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istiod-ca-cert\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.242242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.240858 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-envoy\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.347084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.347049 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-token\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.350226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.348422 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-podinfo\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.368802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.367466 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hvz7\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-kube-api-access-4hvz7\") pod \"e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.370348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.370112 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:05.759470 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:21:05.759441 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2efa0fa_4f83_4432_a568_a5d2ff6d6b54.slice/crio-d3b60527b623af737542b8172da059bf8284102919ac15001f5036b2baded3a6 WatchSource:0}: Error finding container d3b60527b623af737542b8172da059bf8284102919ac15001f5036b2baded3a6: Status 404 returned error can't find the container with id d3b60527b623af737542b8172da059bf8284102919ac15001f5036b2baded3a6 Nov 02 22:21:05.761368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.761337 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz"] Nov 02 22:21:05.763082 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.763044 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:05.763158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.763130 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:05.763195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:05.763174 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:06.612701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:06.610001 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" event={"ID":"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54","Type":"ContainerStarted","Data":"c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742"} Nov 02 22:21:06.612701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:06.610063 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" event={"ID":"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54","Type":"ContainerStarted","Data":"d3b60527b623af737542b8172da059bf8284102919ac15001f5036b2baded3a6"} Nov 02 22:21:06.638884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:06.638809 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" podStartSLOduration=2.6387913210000002 podStartE2EDuration="2.638791321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:06.630015653 +0000 UTC m=+1394.568128872" watchObservedRunningTime="2026-11-02 22:21:06.638791321 +0000 UTC m=+1394.576904539" Nov 02 22:21:07.371146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:07.371113 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:07.379556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:07.379529 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:07.619293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:07.619262 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:07.620185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:07.620161 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:21:09.594569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:09.590544 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:21:09.594569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:09.590596 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 02 22:21:09.645895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:09.645823 4309 generic.go:356] "Generic (PLEG): container finished" podID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerID="4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d" exitCode=0 Nov 02 22:21:09.646045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:09.645903 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" event={"ID":"4b47633d-d531-4e0c-a9ab-2942027ae6cd","Type":"ContainerDied","Data":"4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d"} Nov 02 22:21:09.646481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:09.646462 4309 scope.go:122] "RemoveContainer" containerID="4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d" Nov 02 22:21:09.648067 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:09.648039 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:09.648169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:09.648106 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:09.648169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:09.648136 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:10.071138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.071069 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx"] Nov 02 22:21:10.135698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.129532 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97295b29-c9qzx"] Nov 02 22:21:10.404746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.404214 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa872a23-6787-4da9-a281-93d7057016ef" path="/var/lib/kubelet/pods/fa872a23-6787-4da9-a281-93d7057016ef/volumes" Nov 02 22:21:10.404746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.404668 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2"] Nov 02 22:21:10.415750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.409953 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:10.415750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.410670 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2"] Nov 02 22:21:10.415750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.414593 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5eba37\"" Nov 02 22:21:10.415750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.415304 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5eba37-dockercfg-tf5b4\"" Nov 02 22:21:10.556694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.548624 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4c2g\" (UniqueName: \"kubernetes.io/projected/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-kube-api-access-b4c2g\") pod \"payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2\" (UID: \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:10.556694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.548729 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2\" (UID: \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:10.701342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.696127 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4c2g\" (UniqueName: \"kubernetes.io/projected/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-kube-api-access-b4c2g\") pod \"payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2\" (UID: \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:10.701342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.696255 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2\" (UID: \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:10.701342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.697631 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2\" (UID: \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:10.769849 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.562804 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s"] Nov 02 22:21:10.816840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.804275 4309 generic.go:356] "Generic (PLEG): container finished" podID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerID="372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1" exitCode=0 Nov 02 22:21:10.823827 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.823786 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s"] Nov 02 22:21:10.823997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.823853 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" event={"ID":"b4f752b0-65ba-4b65-b25d-3506c2cf0128","Type":"ContainerDied","Data":"372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1"} Nov 02 22:21:10.824531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.824497 4309 scope.go:122] "RemoveContainer" containerID="372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1" Nov 02 22:21:10.825249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.824505 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:10.828894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.828831 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:10.829146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.829122 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:10.829358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.829319 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:10.860835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.847001 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" event={"ID":"4b47633d-d531-4e0c-a9ab-2942027ae6cd","Type":"ContainerStarted","Data":"f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7"} Nov 02 22:21:10.861924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.861904 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4c2g\" (UniqueName: \"kubernetes.io/projected/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-kube-api-access-b4c2g\") pod \"payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2\" (UID: \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:10.920177 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.920145 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db96d89-7a42-4efb-aad5-f2d19119b488-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s\" (UID: \"7db96d89-7a42-4efb-aad5-f2d19119b488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:10.921261 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:10.920614 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stjzk\" (UniqueName: \"kubernetes.io/projected/7db96d89-7a42-4efb-aad5-f2d19119b488-kube-api-access-stjzk\") pod \"payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s\" (UID: \"7db96d89-7a42-4efb-aad5-f2d19119b488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:11.023242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.023155 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db96d89-7a42-4efb-aad5-f2d19119b488-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s\" (UID: \"7db96d89-7a42-4efb-aad5-f2d19119b488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:11.023429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.023341 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stjzk\" (UniqueName: \"kubernetes.io/projected/7db96d89-7a42-4efb-aad5-f2d19119b488-kube-api-access-stjzk\") pod \"payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s\" (UID: \"7db96d89-7a42-4efb-aad5-f2d19119b488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:11.024047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.024018 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db96d89-7a42-4efb-aad5-f2d19119b488-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s\" (UID: \"7db96d89-7a42-4efb-aad5-f2d19119b488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:11.032563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.032527 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stjzk\" (UniqueName: \"kubernetes.io/projected/7db96d89-7a42-4efb-aad5-f2d19119b488-kube-api-access-stjzk\") pod \"payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s\" (UID: \"7db96d89-7a42-4efb-aad5-f2d19119b488\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:11.104746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.104712 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:11.245847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.220419 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:11.280795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.270273 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:21:11.281452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.281096 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:21:11.281452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.281153 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 02 22:21:11.432624 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:21:11.432453 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb5861e_35ce_4e87_adf8_a1e60203ee3e.slice/crio-7be9650d3391b0805c4d79524195bcc466ae6d6de88a7a6e26060d9686c9a543 WatchSource:0}: Error finding container 7be9650d3391b0805c4d79524195bcc466ae6d6de88a7a6e26060d9686c9a543: Status 404 returned error can't find the container with id 7be9650d3391b0805c4d79524195bcc466ae6d6de88a7a6e26060d9686c9a543 Nov 02 22:21:11.439442 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.439414 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2"] Nov 02 22:21:11.708429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.708384 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s"] Nov 02 22:21:11.742669 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:21:11.742633 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db96d89_7a42_4efb_aad5_f2d19119b488.slice/crio-1a8eafd93d801a2c06e6d6a47811dfa8098f32e07e0d2dd98a865c93b42fcb0c WatchSource:0}: Error finding container 1a8eafd93d801a2c06e6d6a47811dfa8098f32e07e0d2dd98a865c93b42fcb0c: Status 404 returned error can't find the container with id 1a8eafd93d801a2c06e6d6a47811dfa8098f32e07e0d2dd98a865c93b42fcb0c Nov 02 22:21:11.750628 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:21:11.750595 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:21:11.858783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.858709 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" event={"ID":"7db96d89-7a42-4efb-aad5-f2d19119b488","Type":"ContainerStarted","Data":"1a8eafd93d801a2c06e6d6a47811dfa8098f32e07e0d2dd98a865c93b42fcb0c"} Nov 02 22:21:11.861284 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.861242 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" event={"ID":"dbb5861e-35ce-4e87-adf8-a1e60203ee3e","Type":"ContainerStarted","Data":"93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400"} Nov 02 22:21:11.861472 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.861454 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" event={"ID":"dbb5861e-35ce-4e87-adf8-a1e60203ee3e","Type":"ContainerStarted","Data":"7be9650d3391b0805c4d79524195bcc466ae6d6de88a7a6e26060d9686c9a543"} Nov 02 22:21:11.862718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.862660 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:11.867295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.867269 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" event={"ID":"b4f752b0-65ba-4b65-b25d-3506c2cf0128","Type":"ContainerStarted","Data":"d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c"} Nov 02 22:21:11.905699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:11.904822 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" podStartSLOduration=1.904800083 podStartE2EDuration="1.904800083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:11.881100072 +0000 UTC m=+1399.819213288" watchObservedRunningTime="2026-11-02 22:21:11.904800083 +0000 UTC m=+1399.842913303" Nov 02 22:21:12.278855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:12.277959 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:21:12.278855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:12.278029 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 02 22:21:12.656700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:12.653667 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:21:12.735342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:12.735304 4309 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 02 22:21:12.735980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:12.735949 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 02 22:21:12.924757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:12.924418 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" event={"ID":"7db96d89-7a42-4efb-aad5-f2d19119b488","Type":"ContainerStarted","Data":"71da3f4205c4f8042aee8f59609c0f7c2e05ba77b9824dd984ca264a60507c47"} Nov 02 22:21:12.924757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:12.924750 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:12.961982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:12.961697 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" podStartSLOduration=2.961666327 podStartE2EDuration="2.961666327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:12.946847192 +0000 UTC m=+1400.884960412" watchObservedRunningTime="2026-11-02 22:21:12.961666327 +0000 UTC m=+1400.899779567" Nov 02 22:21:13.265954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:13.265848 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:21:13.265954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:13.265902 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 02 22:21:13.635698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:13.624826 4309 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 02 22:21:13.635698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:13.624890 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 02 22:21:14.255699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:14.251120 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:21:14.255699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:14.251187 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 02 22:21:14.621699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:14.620708 4309 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 02 22:21:14.621699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:14.620770 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 02 22:21:14.903757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:14.895766 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:21:15.262793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:15.262091 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:21:15.262793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:15.262156 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 02 22:21:15.619810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:15.614984 4309 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 02 22:21:15.619810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:15.615035 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 02 22:21:16.250298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:16.250252 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:21:16.250494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:16.250354 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 02 22:21:16.613057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:16.613021 4309 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 02 22:21:16.613700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:16.613654 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 02 22:21:17.255309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.254881 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Nov 02 22:21:17.255309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.254942 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Nov 02 22:21:17.623698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.620719 4309 patch_prober.go:29] interesting pod/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Nov 02 22:21:17.623698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.620781 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Nov 02 22:21:17.657698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.653724 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4"] Nov 02 22:21:17.670068 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.670037 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.676764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.676731 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-00c72a-openshift-default-dockercfg-pdxkr\"" Nov 02 22:21:17.853703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.853643 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4d7g\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-kube-api-access-d4d7g\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.855528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.855489 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-envoy\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.855739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.855560 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-credential-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.855739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.855600 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-data\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.855739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.855644 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-certs\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.856056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.855810 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.856056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.855883 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-istio-token\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.856056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.855962 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aeb037-5886-453a-84ea-9cd21986f34a-istiod-ca-cert\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.856056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.856036 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aeb037-5886-453a-84ea-9cd21986f34a-istio-podinfo\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.859467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.858606 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4"] Nov 02 22:21:17.957291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.957176 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-envoy\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.957291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.957250 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-credential-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.957291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.957287 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-data\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.957887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.957333 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-certs\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.957887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.957453 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.957887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.957524 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-istio-token\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.957887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.957614 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aeb037-5886-453a-84ea-9cd21986f34a-istiod-ca-cert\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.957887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.957702 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aeb037-5886-453a-84ea-9cd21986f34a-istio-podinfo\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.957887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.957817 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4d7g\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-kube-api-access-d4d7g\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.959064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.959031 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.959577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.959211 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-data\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.959577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.959460 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-credential-socket\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.959775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.959583 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-certs\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.960274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.960251 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aeb037-5886-453a-84ea-9cd21986f34a-istiod-ca-cert\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.961058 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.961039 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-envoy\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.965213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.965153 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aeb037-5886-453a-84ea-9cd21986f34a-istio-podinfo\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.966715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.966654 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4d7g\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-kube-api-access-d4d7g\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:17.966845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:17.966762 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-istio-token\") pod \"e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:18.156311 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.156277 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:18.259691 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.259597 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:21:18.261757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.261574 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:21:18.263666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.263631 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:21:18.650574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.650542 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4"] Nov 02 22:21:18.692073 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:21:18.692038 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04aeb037_5886_453a_84ea_9cd21986f34a.slice/crio-66ecdefdbd54b32b9a9d7f9080d9ea795e28637f0b915b57fd323cd5e2f25dab WatchSource:0}: Error finding container 66ecdefdbd54b32b9a9d7f9080d9ea795e28637f0b915b57fd323cd5e2f25dab: Status 404 returned error can't find the container with id 66ecdefdbd54b32b9a9d7f9080d9ea795e28637f0b915b57fd323cd5e2f25dab Nov 02 22:21:18.696221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.696151 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:18.696364 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.696293 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:18.696364 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.696356 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:18.738095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.738068 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:21:18.740975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:18.740935 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:21:19.185841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:19.185800 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" event={"ID":"04aeb037-5886-453a-84ea-9cd21986f34a","Type":"ContainerStarted","Data":"40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130"} Nov 02 22:21:19.186116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:19.186088 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" event={"ID":"04aeb037-5886-453a-84ea-9cd21986f34a","Type":"ContainerStarted","Data":"66ecdefdbd54b32b9a9d7f9080d9ea795e28637f0b915b57fd323cd5e2f25dab"} Nov 02 22:21:19.212800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:19.212736 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" podStartSLOduration=2.212717633 podStartE2EDuration="2.212717633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:19.209447324 +0000 UTC m=+1407.147560571" watchObservedRunningTime="2026-11-02 22:21:19.212717633 +0000 UTC m=+1407.150830853" Nov 02 22:21:20.158810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:20.158769 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:20.162655 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:20.159409 4309 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 22:21:20.162655 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:20.159487 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 22:21:21.156778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:21.156742 4309 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 22:21:21.156969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:21.156794 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 22:21:22.157700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:22.157493 4309 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 22:21:22.157700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:22.157586 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 22:21:23.157590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:23.157555 4309 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 22:21:23.157793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:23.157617 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 22:21:23.958707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:23.957952 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:21:24.162064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:24.162030 4309 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 22:21:24.162607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:24.162573 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 22:21:25.037699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:25.036174 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:21:25.189697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:25.158520 4309 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 22:21:25.280894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:25.280858 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 22:21:26.158208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:26.158169 4309 patch_prober.go:29] interesting pod/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 02 22:21:26.158496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:26.158467 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 02 22:21:27.162747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:27.162715 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:27.163270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:27.163251 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:27.164120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:27.164093 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:21:29.443744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.443705 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb"] Nov 02 22:21:29.451203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.451177 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.451851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.451823 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb"] Nov 02 22:21:29.453627 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.453598 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.453988 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.453964 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.454156 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.454138 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625px\" (UniqueName: \"kubernetes.io/projected/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-kube-api-access-625px\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.455682 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.455655 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:29.455780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.455691 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:21:29.455959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.455940 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:21:29.456049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.456033 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:21:29.555721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.555620 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625px\" (UniqueName: \"kubernetes.io/projected/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-kube-api-access-625px\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.555908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.555762 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.556088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.556063 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.556411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.556391 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.558539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.558516 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.564014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.563992 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-625px\" (UniqueName: \"kubernetes.io/projected/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-kube-api-access-625px\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:29.771762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:29.771654 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:30.237052 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:30.237010 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb"] Nov 02 22:21:30.244215 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:21:30.244179 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4df277d_e9d4_43c4_a08e_b37a0d51bcc2.slice/crio-c07eac05439a73a53ecba821e88a373ab4d85c59d6affcfd273dedd5f6b3b165 WatchSource:0}: Error finding container c07eac05439a73a53ecba821e88a373ab4d85c59d6affcfd273dedd5f6b3b165: Status 404 returned error can't find the container with id c07eac05439a73a53ecba821e88a373ab4d85c59d6affcfd273dedd5f6b3b165 Nov 02 22:21:30.765931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:30.765885 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" event={"ID":"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2","Type":"ContainerStarted","Data":"c07eac05439a73a53ecba821e88a373ab4d85c59d6affcfd273dedd5f6b3b165"} Nov 02 22:21:31.776043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:31.775961 4309 generic.go:356] "Generic (PLEG): container finished" podID="e4df277d-e9d4-43c4-a08e-b37a0d51bcc2" containerID="f8fd4f230d79a660671807852b75334fb035efd2ec2d80380843b54f747a687b" exitCode=0 Nov 02 22:21:31.776418 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:31.776039 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" event={"ID":"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2","Type":"ContainerDied","Data":"f8fd4f230d79a660671807852b75334fb035efd2ec2d80380843b54f747a687b"} Nov 02 22:21:32.825838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:32.825794 4309 patch_prober.go:29] interesting pod/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:21:32.826306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:32.825869 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:21:32.998438 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:32.998405 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:33.101322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.099669 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-xks-ca\") pod \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " Nov 02 22:21:33.101322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.099777 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-openshift-ca\") pod \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " Nov 02 22:21:33.101322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.099932 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-kube-api-access-625px\" (UniqueName: \"kubernetes.io/projected/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-kube-api-access-625px\") pod \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\" (UID: \"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2\") " Nov 02 22:21:33.102465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.102441 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-openshift-ca" pod "e4df277d-e9d4-43c4-a08e-b37a0d51bcc2" (UID: "e4df277d-e9d4-43c4-a08e-b37a0d51bcc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:21:33.134224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.134162 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-xks-ca" pod "e4df277d-e9d4-43c4-a08e-b37a0d51bcc2" (UID: "e4df277d-e9d4-43c4-a08e-b37a0d51bcc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:21:33.143712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.139779 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-kube-api-access-625px" pod "e4df277d-e9d4-43c4-a08e-b37a0d51bcc2" (UID: "e4df277d-e9d4-43c4-a08e-b37a0d51bcc2"). InnerVolumeSpecName "kube-api-access-625px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:21:33.203933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.203846 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-625px\" (UniqueName: \"kubernetes.io/projected/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-kube-api-access-625px\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:33.203933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.203887 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:33.203933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.203905 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:21:33.821919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.821880 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" event={"ID":"e4df277d-e9d4-43c4-a08e-b37a0d51bcc2","Type":"ContainerDied","Data":"c07eac05439a73a53ecba821e88a373ab4d85c59d6affcfd273dedd5f6b3b165"} Nov 02 22:21:33.822101 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.821937 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c07eac05439a73a53ecba821e88a373ab4d85c59d6affcfd273dedd5f6b3b165" Nov 02 22:21:33.822101 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:33.821972 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb" Nov 02 22:21:35.991076 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:21:35.991041 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@10-31002-0.service\": RecentStats: unable to find data in memory cache]" Nov 02 22:21:36.745352 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.745314 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc"] Nov 02 22:21:36.746791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.746763 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e4df277d-e9d4-43c4-a08e-b37a0d51bcc2" containerName="revoke-keys" Nov 02 22:21:36.746791 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.746789 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4df277d-e9d4-43c4-a08e-b37a0d51bcc2" containerName="revoke-keys" Nov 02 22:21:36.747069 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.747055 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4df277d-e9d4-43c4-a08e-b37a0d51bcc2" containerName="revoke-keys" Nov 02 22:21:36.751002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.750986 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:36.760095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.760071 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc"] Nov 02 22:21:36.765197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.765172 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l"] Nov 02 22:21:36.777016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.776992 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:36.792953 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.792911 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l"] Nov 02 22:21:36.851936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.851905 4309 generic.go:356] "Generic (PLEG): container finished" podID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerID="1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962" exitCode=0 Nov 02 22:21:36.852128 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.851979 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" event={"ID":"8e3c8d49-94e7-402f-945c-2888cf3a8930","Type":"ContainerDied","Data":"1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962"} Nov 02 22:21:36.852515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.852498 4309 scope.go:122] "RemoveContainer" containerID="1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962" Nov 02 22:21:36.854476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.854447 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:36.854560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.854514 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:36.854560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.854549 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:36.985607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.982067 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:36.985607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:36.982144 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.022568 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.022660 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.022705 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5f2\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-kube-api-access-mf5f2\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.022795 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.022837 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.022859 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.022888 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.022935 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.022990 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.023019 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.023134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.023091 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.034391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.024100 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.034391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.024185 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.034391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.024490 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.034391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.024571 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.034391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.024739 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7w9j\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-kube-api-access-j7w9j\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.126505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.126468 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7w9j\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-kube-api-access-j7w9j\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.126709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.126523 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.126709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.126554 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.126709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.126590 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.126709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.126623 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.126995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.126809 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5f2\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-kube-api-access-mf5f2\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.126995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.126953 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.126995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.126991 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.127228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127002 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.127228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127015 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.127228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127059 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.127228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127102 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.127228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127144 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.127228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127169 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.127228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127191 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.127610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127200 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.127610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127267 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.127610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127331 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.127610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127401 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.127610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127428 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.127610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127502 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.127610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127521 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-credential-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.127610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127576 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.128045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127806 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-certs\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.128045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.127934 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-data\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.131999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.128460 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.131999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.128961 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istiod-ca-cert\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.131999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.129172 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-socket\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.136571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.136303 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5f2\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-kube-api-access-mf5f2\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.136571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.136519 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.137204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.137182 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.137367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.137344 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7w9j\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-kube-api-access-j7w9j\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.137453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.137355 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-envoy\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.137821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.137802 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.137911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.137825 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-token\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.138275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.138258 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-podinfo\") pod \"e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.375475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.375437 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:37.420361 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.420318 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:37.710397 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:21:37.710348 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4b5c51_e840_421e_8353_f1f3b7a604f1.slice/crio-8127adb0467cd1d524ac3f36b7c1c30891d62154699fc0e6ae23958dc4861cad WatchSource:0}: Error finding container 8127adb0467cd1d524ac3f36b7c1c30891d62154699fc0e6ae23958dc4861cad: Status 404 returned error can't find the container with id 8127adb0467cd1d524ac3f36b7c1c30891d62154699fc0e6ae23958dc4861cad Nov 02 22:21:37.711498 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.711470 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc"] Nov 02 22:21:37.716737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.715399 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:37.716737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.715486 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:37.716737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.715546 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:37.865726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.864668 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" event={"ID":"8e3c8d49-94e7-402f-945c-2888cf3a8930","Type":"ContainerStarted","Data":"db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916"} Nov 02 22:21:37.866870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.866845 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" event={"ID":"dc4b5c51-e840-421e-8353-f1f3b7a604f1","Type":"ContainerStarted","Data":"8127adb0467cd1d524ac3f36b7c1c30891d62154699fc0e6ae23958dc4861cad"} Nov 02 22:21:37.948937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.945483 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l"] Nov 02 22:21:37.952317 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:21:37.949416 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b69bd1d_28c3_4c79_8a0f_4442d80b1691.slice/crio-504e887202d24208b504dc8ea206f91f262c9363882bcd445a6d829c6d2e114b WatchSource:0}: Error finding container 504e887202d24208b504dc8ea206f91f262c9363882bcd445a6d829c6d2e114b: Status 404 returned error can't find the container with id 504e887202d24208b504dc8ea206f91f262c9363882bcd445a6d829c6d2e114b Nov 02 22:21:37.954169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.954133 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:37.954236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.954221 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:37.954280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:37.954263 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:21:38.564699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:38.563662 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:21:38.564699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:38.563856 4309 patch_prober.go:29] interesting pod/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 22:21:38.564699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:38.563903 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 22:21:38.913757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:38.912798 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" event={"ID":"4b69bd1d-28c3-4c79-8a0f-4442d80b1691","Type":"ContainerStarted","Data":"ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed"} Nov 02 22:21:38.913757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:38.912872 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" event={"ID":"4b69bd1d-28c3-4c79-8a0f-4442d80b1691","Type":"ContainerStarted","Data":"504e887202d24208b504dc8ea206f91f262c9363882bcd445a6d829c6d2e114b"} Nov 02 22:21:38.941698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:38.941230 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" event={"ID":"dc4b5c51-e840-421e-8353-f1f3b7a604f1","Type":"ContainerStarted","Data":"5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f"} Nov 02 22:21:38.973706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:38.960912 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podStartSLOduration=2.9608926049999997 podStartE2EDuration="2.960892605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:38.950127352 +0000 UTC m=+1426.888240570" watchObservedRunningTime="2026-11-02 22:21:38.960892605 +0000 UTC m=+1426.899005824" Nov 02 22:21:39.036699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.035708 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podStartSLOduration=3.035689232 podStartE2EDuration="3.035689232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:21:38.986108952 +0000 UTC m=+1426.924222172" watchObservedRunningTime="2026-11-02 22:21:39.035689232 +0000 UTC m=+1426.973802451" Nov 02 22:21:39.381701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.380991 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:39.385697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.384727 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:21:39.385697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.384786 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:21:39.426695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.426482 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:39.426695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.426649 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 22:21:39.426953 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.426712 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 22:21:39.577253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.577217 4309 patch_prober.go:29] interesting pod/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 22:21:39.577873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.577841 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 22:21:39.974825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:39.974767 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:21:40.522933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:40.089850 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:21:40.522933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:40.475419 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 22:21:40.522933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:40.518992 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 22:21:40.522933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:40.475743 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:21:40.522933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:40.519040 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:21:40.597826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:40.594339 4309 patch_prober.go:29] interesting pod/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 22:21:40.597826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:40.594396 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 22:21:41.132699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:41.131892 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:21:41.161781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:41.157045 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 22:21:41.401063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:41.400963 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:21:41.407840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:41.407785 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:21:41.443373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:41.443335 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 22:21:41.467905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:41.467863 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 22:21:41.572391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:41.572347 4309 patch_prober.go:29] interesting pod/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 22:21:41.572714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:41.572658 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 22:21:41.729473 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:21:41.729373 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:21:42.381618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:42.381569 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:21:42.407573 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:42.381630 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:21:42.432700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:42.432099 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 22:21:42.432700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:42.432160 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 22:21:42.564580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:42.564540 4309 patch_prober.go:29] interesting pod/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 22:21:42.564934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:42.564906 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 22:21:43.103739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:43.099612 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9"] Nov 02 22:21:43.103739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:43.099699 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25e1cfae-bfvb9"] Nov 02 22:21:43.377436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:43.377345 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:21:43.377742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:43.377712 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:21:43.446726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:43.446670 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 22:21:43.450787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:43.450745 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 22:21:43.639751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:43.638861 4309 patch_prober.go:29] interesting pod/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 22:21:43.639751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:43.638940 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 22:21:44.142984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.142937 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:21:44.185699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.171774 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:21:44.247588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.247543 4309 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:21:44.256714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.256110 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e05464-47f8-4403-9d22-4cd6091c5b90" path="/var/lib/kubelet/pods/48e05464-47f8-4403-9d22-4cd6091c5b90/volumes" Nov 02 22:21:44.398761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.377038 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:21:44.408298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.407821 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:21:44.428766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.421641 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 22:21:44.428766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.421716 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 22:21:44.587799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.576210 4309 patch_prober.go:29] interesting pod/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 02 22:21:44.587799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:44.576269 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 02 22:21:45.376542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:45.376135 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:21:45.376542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:45.376196 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:21:45.446703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:45.424359 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 22:21:45.446703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:45.424417 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 22:21:46.160561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:46.160528 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:21:46.163174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:46.163154 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:21:46.169144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:46.169119 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:21:46.377697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:46.377230 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Nov 02 22:21:46.377697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:46.377292 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Nov 02 22:21:46.422751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:46.422101 4309 patch_prober.go:29] interesting pod/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 02 22:21:46.422751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:46.422159 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 02 22:21:47.381979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:47.381949 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:47.382516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:47.382494 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:47.383312 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:47.383288 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:21:47.428144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:47.428111 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:47.428987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:47.428963 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:47.429867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:47.429840 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:21:51.166855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:51.166804 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb"] Nov 02 22:21:51.174665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:51.174611 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7144fe-p9cjb"] Nov 02 22:21:52.192512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:52.192479 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4df277d-e9d4-43c4-a08e-b37a0d51bcc2" path="/var/lib/kubelet/pods/e4df277d-e9d4-43c4-a08e-b37a0d51bcc2/volumes" Nov 02 22:21:56.023902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:56.023864 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg"] Nov 02 22:21:56.028353 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:56.028322 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e41425e-zgbhg"] Nov 02 22:21:56.194131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:56.194096 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c914f992-2af7-4a09-8c7c-940c4ceac741" path="/var/lib/kubelet/pods/c914f992-2af7-4a09-8c7c-940c4ceac741/volumes" Nov 02 22:21:59.422015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.419824 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99"] Nov 02 22:21:59.435164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.428916 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99"] Nov 02 22:21:59.435164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.429042 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.437934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.437913 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:21:59.438175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.438161 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:21:59.499061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.499031 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:21:59.499259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.499101 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:21:59.531029 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.530991 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.531257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.531240 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.531463 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.531442 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbf4\" (UniqueName: \"kubernetes.io/projected/f2a89735-0efa-4e95-a377-c45f73c7cb24-kube-api-access-ffbf4\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.633365 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.633329 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.633544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.633513 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbf4\" (UniqueName: \"kubernetes.io/projected/f2a89735-0efa-4e95-a377-c45f73c7cb24-kube-api-access-ffbf4\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.633773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.633737 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.634349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.634324 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.635843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.635825 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.641302 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.641275 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbf4\" (UniqueName: \"kubernetes.io/projected/f2a89735-0efa-4e95-a377-c45f73c7cb24-kube-api-access-ffbf4\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:21:59.754639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:21:59.754559 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:22:00.053066 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:22:00.053036 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a89735_0efa_4e95_a377_c45f73c7cb24.slice/crio-6b7e62dae8506cb73e021f308258f9b3c8fabb8c896fa636fabb9a7a1771bf5b WatchSource:0}: Error finding container 6b7e62dae8506cb73e021f308258f9b3c8fabb8c896fa636fabb9a7a1771bf5b: Status 404 returned error can't find the container with id 6b7e62dae8506cb73e021f308258f9b3c8fabb8c896fa636fabb9a7a1771bf5b Nov 02 22:22:00.053782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:00.053640 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99"] Nov 02 22:22:00.354893 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:00.354857 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" event={"ID":"f2a89735-0efa-4e95-a377-c45f73c7cb24","Type":"ContainerStarted","Data":"6b7e62dae8506cb73e021f308258f9b3c8fabb8c896fa636fabb9a7a1771bf5b"} Nov 02 22:22:01.364557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:01.364476 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2a89735-0efa-4e95-a377-c45f73c7cb24" containerID="e3ac785be847cee2583b7128bb2bd938b97bedf4db189f421ef86fa08f306291" exitCode=0 Nov 02 22:22:01.364960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:01.364557 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" event={"ID":"f2a89735-0efa-4e95-a377-c45f73c7cb24","Type":"ContainerDied","Data":"e3ac785be847cee2583b7128bb2bd938b97bedf4db189f421ef86fa08f306291"} Nov 02 22:22:02.582860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.582826 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:22:02.709244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.709162 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-xks-ca\") pod \"f2a89735-0efa-4e95-a377-c45f73c7cb24\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " Nov 02 22:22:02.709403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.709284 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a89735-0efa-4e95-a377-c45f73c7cb24-kube-api-access-ffbf4\" (UniqueName: \"kubernetes.io/projected/f2a89735-0efa-4e95-a377-c45f73c7cb24-kube-api-access-ffbf4\") pod \"f2a89735-0efa-4e95-a377-c45f73c7cb24\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " Nov 02 22:22:02.709403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.709370 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-openshift-ca\") pod \"f2a89735-0efa-4e95-a377-c45f73c7cb24\" (UID: \"f2a89735-0efa-4e95-a377-c45f73c7cb24\") " Nov 02 22:22:02.709858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.709825 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-openshift-ca" pod "f2a89735-0efa-4e95-a377-c45f73c7cb24" (UID: "f2a89735-0efa-4e95-a377-c45f73c7cb24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:02.710442 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.710416 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:02.711944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.711916 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-xks-ca" pod "f2a89735-0efa-4e95-a377-c45f73c7cb24" (UID: "f2a89735-0efa-4e95-a377-c45f73c7cb24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:02.712047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.711978 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a89735-0efa-4e95-a377-c45f73c7cb24-kube-api-access-ffbf4" pod "f2a89735-0efa-4e95-a377-c45f73c7cb24" (UID: "f2a89735-0efa-4e95-a377-c45f73c7cb24"). InnerVolumeSpecName "kube-api-access-ffbf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:02.812836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.812798 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2a89735-0efa-4e95-a377-c45f73c7cb24-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:02.812836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:02.812836 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffbf4\" (UniqueName: \"kubernetes.io/projected/f2a89735-0efa-4e95-a377-c45f73c7cb24-kube-api-access-ffbf4\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:03.418827 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:03.418787 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" event={"ID":"f2a89735-0efa-4e95-a377-c45f73c7cb24","Type":"ContainerDied","Data":"6b7e62dae8506cb73e021f308258f9b3c8fabb8c896fa636fabb9a7a1771bf5b"} Nov 02 22:22:03.425733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:03.425693 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b7e62dae8506cb73e021f308258f9b3c8fabb8c896fa636fabb9a7a1771bf5b" Nov 02 22:22:03.425733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:03.419158 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99" Nov 02 22:22:11.602510 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:22:11.602405 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:22:19.200398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:19.200363 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99"] Nov 02 22:22:19.204752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:19.204733 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-334860-9vd99"] Nov 02 22:22:20.193153 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:20.193117 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a89735-0efa-4e95-a377-c45f73c7cb24" path="/var/lib/kubelet/pods/f2a89735-0efa-4e95-a377-c45f73c7cb24/volumes" Nov 02 22:22:27.915632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.915599 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz"] Nov 02 22:22:27.916657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.916636 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2a89735-0efa-4e95-a377-c45f73c7cb24" containerName="revoke-keys" Nov 02 22:22:27.916657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.916657 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a89735-0efa-4e95-a377-c45f73c7cb24" containerName="revoke-keys" Nov 02 22:22:27.916794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.916785 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a89735-0efa-4e95-a377-c45f73c7cb24" containerName="revoke-keys" Nov 02 22:22:27.921344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.921314 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:27.924643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.924603 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:22:27.926065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.926041 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:22:27.926187 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.926064 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:22:27.926187 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.926154 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:22:27.926746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:27.926723 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz"] Nov 02 22:22:28.069697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.062927 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.069697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.063153 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.069697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.063363 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hzz\" (UniqueName: \"kubernetes.io/projected/65c91168-06a6-4f02-b2b0-48924b4cb525-kube-api-access-l9hzz\") pod \"maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.164793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.164764 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hzz\" (UniqueName: \"kubernetes.io/projected/65c91168-06a6-4f02-b2b0-48924b4cb525-kube-api-access-l9hzz\") pod \"maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.164985 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.164910 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.165102 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.165087 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.165802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.165737 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.167299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.167276 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.172805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.172782 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hzz\" (UniqueName: \"kubernetes.io/projected/65c91168-06a6-4f02-b2b0-48924b4cb525-kube-api-access-l9hzz\") pod \"maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.240601 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.240573 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:28.527550 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:22:28.527515 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c91168_06a6_4f02_b2b0_48924b4cb525.slice/crio-3984c2416e962800235491face56d23466588a61e9bfddc3c19fab7ff3a77ffa WatchSource:0}: Error finding container 3984c2416e962800235491face56d23466588a61e9bfddc3c19fab7ff3a77ffa: Status 404 returned error can't find the container with id 3984c2416e962800235491face56d23466588a61e9bfddc3c19fab7ff3a77ffa Nov 02 22:22:28.530080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.530049 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz"] Nov 02 22:22:28.943022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:28.942972 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" event={"ID":"65c91168-06a6-4f02-b2b0-48924b4cb525","Type":"ContainerStarted","Data":"3984c2416e962800235491face56d23466588a61e9bfddc3c19fab7ff3a77ffa"} Nov 02 22:22:29.952890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:29.952806 4309 generic.go:356] "Generic (PLEG): container finished" podID="65c91168-06a6-4f02-b2b0-48924b4cb525" containerID="4a1fd264b475d6736f0510f5f2f1d213357db4a0d1ee08cecf53ff1772c368e3" exitCode=0 Nov 02 22:22:29.953269 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:29.952992 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" event={"ID":"65c91168-06a6-4f02-b2b0-48924b4cb525","Type":"ContainerDied","Data":"4a1fd264b475d6736f0510f5f2f1d213357db4a0d1ee08cecf53ff1772c368e3"} Nov 02 22:22:30.607706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.607636 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww"] Nov 02 22:22:30.612571 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.612545 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.619266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.619240 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d28bd1-openshift-default-dockercfg-9p2fd\"" Nov 02 22:22:30.626044 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.626019 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww"] Nov 02 22:22:30.767957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.753736 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-envoy\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.767957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.753896 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-credential-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.767957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.754017 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-token\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.767957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.754060 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-data\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.767957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.754099 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istiod-ca-cert\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.767957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.754231 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.767957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.754290 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2lkb\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-kube-api-access-h2lkb\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.767957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.754365 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-certs\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.767957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.754406 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-podinfo\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.855972 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.855937 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-token\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.855985 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-data\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856011 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istiod-ca-cert\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856088 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856127 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2lkb\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-kube-api-access-h2lkb\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856161 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-certs\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856187 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-podinfo\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856346 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-envoy\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856351 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-data\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856477 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-credential-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856518 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856824 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istiod-ca-cert\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856949 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856832 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-credential-socket\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.856949 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.856910 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-certs\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.858649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.858600 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-envoy\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.858964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.858942 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-podinfo\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.865119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.865097 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-token\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.865292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.865268 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2lkb\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-kube-api-access-h2lkb\") pod \"e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:30.937599 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:30.937569 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:31.207874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.207522 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:31.241310 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:22:31.241276 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfc3202_ffd9_4470_b3bb_f0ced5d8311f.slice/crio-d9f17cebd6d15e187098ca32f57818c51225367294022f80e41ad06d1f5cb145 WatchSource:0}: Error finding container d9f17cebd6d15e187098ca32f57818c51225367294022f80e41ad06d1f5cb145: Status 404 returned error can't find the container with id d9f17cebd6d15e187098ca32f57818c51225367294022f80e41ad06d1f5cb145 Nov 02 22:22:31.241561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.241525 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww"] Nov 02 22:22:31.244668 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.244631 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:22:31.244798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.244764 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:22:31.244861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.244807 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:22:31.364764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.364733 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-xks-ca\") pod \"65c91168-06a6-4f02-b2b0-48924b4cb525\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " Nov 02 22:22:31.366044 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.366021 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-openshift-ca\") pod \"65c91168-06a6-4f02-b2b0-48924b4cb525\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " Nov 02 22:22:31.366223 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.366199 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c91168-06a6-4f02-b2b0-48924b4cb525-kube-api-access-l9hzz\" (UniqueName: \"kubernetes.io/projected/65c91168-06a6-4f02-b2b0-48924b4cb525-kube-api-access-l9hzz\") pod \"65c91168-06a6-4f02-b2b0-48924b4cb525\" (UID: \"65c91168-06a6-4f02-b2b0-48924b4cb525\") " Nov 02 22:22:31.366481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.366455 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-openshift-ca" pod "65c91168-06a6-4f02-b2b0-48924b4cb525" (UID: "65c91168-06a6-4f02-b2b0-48924b4cb525"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:31.367176 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.367150 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:31.367347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.367327 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-xks-ca" pod "65c91168-06a6-4f02-b2b0-48924b4cb525" (UID: "65c91168-06a6-4f02-b2b0-48924b4cb525"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:31.368600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.368574 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c91168-06a6-4f02-b2b0-48924b4cb525-kube-api-access-l9hzz" pod "65c91168-06a6-4f02-b2b0-48924b4cb525" (UID: "65c91168-06a6-4f02-b2b0-48924b4cb525"). InnerVolumeSpecName "kube-api-access-l9hzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:31.511665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.511568 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9hzz\" (UniqueName: \"kubernetes.io/projected/65c91168-06a6-4f02-b2b0-48924b4cb525-kube-api-access-l9hzz\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:31.511928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.511905 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c91168-06a6-4f02-b2b0-48924b4cb525-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:31.975959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.975915 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" event={"ID":"adfc3202-ffd9-4470-b3bb-f0ced5d8311f","Type":"ContainerStarted","Data":"68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e"} Nov 02 22:22:31.976159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.975986 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" event={"ID":"adfc3202-ffd9-4470-b3bb-f0ced5d8311f","Type":"ContainerStarted","Data":"d9f17cebd6d15e187098ca32f57818c51225367294022f80e41ad06d1f5cb145"} Nov 02 22:22:31.977538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.977504 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" Nov 02 22:22:31.977655 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.977551 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz" event={"ID":"65c91168-06a6-4f02-b2b0-48924b4cb525","Type":"ContainerDied","Data":"3984c2416e962800235491face56d23466588a61e9bfddc3c19fab7ff3a77ffa"} Nov 02 22:22:31.977655 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:31.977591 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3984c2416e962800235491face56d23466588a61e9bfddc3c19fab7ff3a77ffa" Nov 02 22:22:32.000627 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:32.000575 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" podStartSLOduration=2.000561191 podStartE2EDuration="2.000561191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:31.998416666 +0000 UTC m=+1479.936529885" watchObservedRunningTime="2026-11-02 22:22:32.000561191 +0000 UTC m=+1479.938674485" Nov 02 22:22:32.717774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:32.717733 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2"] Nov 02 22:22:32.736584 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:32.736549 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2"] Nov 02 22:22:32.940850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:32.940816 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:32.944711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:32.941590 4309 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:22:32.944711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:32.941657 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:22:33.938800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:33.938765 4309 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:22:33.939203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:33.938822 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:22:34.025592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:34.025558 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh"] Nov 02 22:22:34.030151 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:34.030121 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07063a45-zzvwh"] Nov 02 22:22:34.285887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:34.285799 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d92fa8-1047-4e44-b0d5-c0358f907f7f" path="/var/lib/kubelet/pods/d7d92fa8-1047-4e44-b0d5-c0358f907f7f/volumes" Nov 02 22:22:34.945699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:34.941228 4309 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:22:34.945699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:34.941302 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:22:35.937960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:35.937919 4309 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:22:35.938152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:35.937988 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:22:36.938204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:36.938171 4309 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:22:36.938599 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:36.938226 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:22:37.947640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:37.938835 4309 patch_prober.go:29] interesting pod/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 02 22:22:37.947640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:37.938896 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 02 22:22:38.178761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:38.178535 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6"] Nov 02 22:22:39.012070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:39.010001 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:39.012070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:39.011871 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:39.014429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:39.014403 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:22:41.646973 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:22:41.646929 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:22:43.179789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:43.179746 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" podUID="1292bd76-2ee4-43b5-b3fe-f77417b25e30" containerName="payload-processing" containerID="cri-o://34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063" gracePeriod=25 Nov 02 22:22:47.757557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:47.757475 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" containerID="cri-o://b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc" gracePeriod=45 Nov 02 22:22:47.761399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:47.761372 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" containerID="cri-o://a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c" gracePeriod=45 Nov 02 22:22:48.122584 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.121879 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:22:48.127795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.127714 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:22:48.197166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197130 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-model-cache\") pod \"ae16f350-d7fa-44dd-bdff-4493c75156b8\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " Nov 02 22:22:48.197338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197219 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-model-cache\") pod \"edf36169-2951-4b4a-a8bc-20b19196adaf\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " Nov 02 22:22:48.197338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197301 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-dshm\") pod \"ae16f350-d7fa-44dd-bdff-4493c75156b8\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " Nov 02 22:22:48.197472 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197347 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-dshm\") pod \"edf36169-2951-4b4a-a8bc-20b19196adaf\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " Nov 02 22:22:48.197472 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197377 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-tmp-dir\") pod \"ae16f350-d7fa-44dd-bdff-4493c75156b8\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " Nov 02 22:22:48.197472 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197417 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-home\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-home\") pod \"edf36169-2951-4b4a-a8bc-20b19196adaf\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " Nov 02 22:22:48.197472 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197454 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf36169-2951-4b4a-a8bc-20b19196adaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/edf36169-2951-4b4a-a8bc-20b19196adaf-tls-certs\") pod \"edf36169-2951-4b4a-a8bc-20b19196adaf\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " Nov 02 22:22:48.197713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197509 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf36169-2951-4b4a-a8bc-20b19196adaf-kube-api-access-pv765\" (UniqueName: \"kubernetes.io/projected/edf36169-2951-4b4a-a8bc-20b19196adaf-kube-api-access-pv765\") pod \"edf36169-2951-4b4a-a8bc-20b19196adaf\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " Nov 02 22:22:48.197713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197561 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-tmp-dir\") pod \"edf36169-2951-4b4a-a8bc-20b19196adaf\" (UID: \"edf36169-2951-4b4a-a8bc-20b19196adaf\") " Nov 02 22:22:48.197713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197651 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae16f350-d7fa-44dd-bdff-4493c75156b8-kube-api-access-wt92k\" (UniqueName: \"kubernetes.io/projected/ae16f350-d7fa-44dd-bdff-4493c75156b8-kube-api-access-wt92k\") pod \"ae16f350-d7fa-44dd-bdff-4493c75156b8\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " Nov 02 22:22:48.197948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197783 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae16f350-d7fa-44dd-bdff-4493c75156b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae16f350-d7fa-44dd-bdff-4493c75156b8-tls-certs\") pod \"ae16f350-d7fa-44dd-bdff-4493c75156b8\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " Nov 02 22:22:48.197948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.197858 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-home\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-home\") pod \"ae16f350-d7fa-44dd-bdff-4493c75156b8\" (UID: \"ae16f350-d7fa-44dd-bdff-4493c75156b8\") " Nov 02 22:22:48.198999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.198967 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-home" pod "ae16f350-d7fa-44dd-bdff-4493c75156b8" (UID: "ae16f350-d7fa-44dd-bdff-4493c75156b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:48.210816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.203095 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-model-cache" pod "ae16f350-d7fa-44dd-bdff-4493c75156b8" (UID: "ae16f350-d7fa-44dd-bdff-4493c75156b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:48.210816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.203312 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-model-cache" pod "edf36169-2951-4b4a-a8bc-20b19196adaf" (UID: "edf36169-2951-4b4a-a8bc-20b19196adaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:48.210816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.204490 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-tmp-dir" pod "ae16f350-d7fa-44dd-bdff-4493c75156b8" (UID: "ae16f350-d7fa-44dd-bdff-4493c75156b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:48.210816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.204722 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-tmp-dir" pod "edf36169-2951-4b4a-a8bc-20b19196adaf" (UID: "edf36169-2951-4b4a-a8bc-20b19196adaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:48.210816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.205818 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-home" pod "edf36169-2951-4b4a-a8bc-20b19196adaf" (UID: "edf36169-2951-4b4a-a8bc-20b19196adaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:48.210816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.207372 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-dshm" pod "edf36169-2951-4b4a-a8bc-20b19196adaf" (UID: "edf36169-2951-4b4a-a8bc-20b19196adaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:48.210816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.208386 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae16f350-d7fa-44dd-bdff-4493c75156b8-tls-certs" pod "ae16f350-d7fa-44dd-bdff-4493c75156b8" (UID: "ae16f350-d7fa-44dd-bdff-4493c75156b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:48.210816 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.209784 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf36169-2951-4b4a-a8bc-20b19196adaf-kube-api-access-pv765" pod "edf36169-2951-4b4a-a8bc-20b19196adaf" (UID: "edf36169-2951-4b4a-a8bc-20b19196adaf"). InnerVolumeSpecName "kube-api-access-pv765". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:48.212590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.212550 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-dshm" pod "ae16f350-d7fa-44dd-bdff-4493c75156b8" (UID: "ae16f350-d7fa-44dd-bdff-4493c75156b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:22:48.212804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.212638 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf36169-2951-4b4a-a8bc-20b19196adaf-tls-certs" pod "edf36169-2951-4b4a-a8bc-20b19196adaf" (UID: "edf36169-2951-4b4a-a8bc-20b19196adaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:48.212922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.212851 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae16f350-d7fa-44dd-bdff-4493c75156b8-kube-api-access-wt92k" pod "ae16f350-d7fa-44dd-bdff-4493c75156b8" (UID: "ae16f350-d7fa-44dd-bdff-4493c75156b8"). InnerVolumeSpecName "kube-api-access-wt92k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341287 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341324 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341339 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341352 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341393 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341406 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341418 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf36169-2951-4b4a-a8bc-20b19196adaf-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341435 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv765\" (UniqueName: \"kubernetes.io/projected/edf36169-2951-4b4a-a8bc-20b19196adaf-kube-api-access-pv765\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341449 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf36169-2951-4b4a-a8bc-20b19196adaf-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341463 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt92k\" (UniqueName: \"kubernetes.io/projected/ae16f350-d7fa-44dd-bdff-4493c75156b8-kube-api-access-wt92k\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341476 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae16f350-d7fa-44dd-bdff-4493c75156b8-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.341581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.341489 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae16f350-d7fa-44dd-bdff-4493c75156b8-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:48.661214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.661124 4309 generic.go:356] "Generic (PLEG): container finished" podID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerID="b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc" exitCode=0 Nov 02 22:22:48.661214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.661166 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" Nov 02 22:22:48.661452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.661221 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" event={"ID":"ae16f350-d7fa-44dd-bdff-4493c75156b8","Type":"ContainerDied","Data":"b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc"} Nov 02 22:22:48.661452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.661270 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2" event={"ID":"ae16f350-d7fa-44dd-bdff-4493c75156b8","Type":"ContainerDied","Data":"8b13f14eb9ab0021a20dc2547d0aa98af182c1c56e9f79f8b5ee4f73f05cf1b1"} Nov 02 22:22:48.661452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.661292 4309 scope.go:122] "RemoveContainer" containerID="b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc" Nov 02 22:22:48.663104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.663070 4309 generic.go:356] "Generic (PLEG): container finished" podID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerID="a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c" exitCode=0 Nov 02 22:22:48.663216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.663125 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" Nov 02 22:22:48.663216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.663182 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" event={"ID":"edf36169-2951-4b4a-a8bc-20b19196adaf","Type":"ContainerDied","Data":"a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c"} Nov 02 22:22:48.663216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.663213 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" event={"ID":"edf36169-2951-4b4a-a8bc-20b19196adaf","Type":"ContainerDied","Data":"8a7e188d8641123f9d8e173f6e8ba0a02199d3bf47679d4399d8b3cb17e57995"} Nov 02 22:22:48.675368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.675349 4309 scope.go:122] "RemoveContainer" containerID="b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc" Nov 02 22:22:48.675982 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:22:48.675723 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc\": container with ID starting with b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc not found: ID does not exist" containerID="b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc" Nov 02 22:22:48.675982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.675757 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc"} err="failed to get container status \"b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc\": rpc error: code = NotFound desc = could not find container \"b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc\": container with ID starting with b895f3b48e3d723e5fe797aebaccc07128cbeda4aebb6b4e02a27f01a9a409dc not found: ID does not exist" Nov 02 22:22:48.675982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.675777 4309 scope.go:122] "RemoveContainer" containerID="a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c" Nov 02 22:22:48.687274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.687253 4309 scope.go:122] "RemoveContainer" containerID="a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c" Nov 02 22:22:48.687594 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:22:48.687570 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c\": container with ID starting with a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c not found: ID does not exist" containerID="a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c" Nov 02 22:22:48.687705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.687601 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c"} err="failed to get container status \"a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c\": rpc error: code = NotFound desc = could not find container \"a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c\": container with ID starting with a8ddc05190d999220a73b0829b8b20392003be3b05566c7067e419e0e3ce2e7c not found: ID does not exist" Nov 02 22:22:48.691647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.691628 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2"] Nov 02 22:22:48.696332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.696311 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-2-w4-c302b6-kserve-7cbd578554-8dtb2"] Nov 02 22:22:48.705910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.705888 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2"] Nov 02 22:22:48.710271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:48.710250 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2"] Nov 02 22:22:49.104018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:49.103979 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c302b6/distinct-w4-c302b6-kserve-775b8f776f-px9c2" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:22:50.035547 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.035505 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz"] Nov 02 22:22:50.043074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.043049 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9d0683-fvjcz"] Nov 02 22:22:50.127696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.126981 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh"] Nov 02 22:22:50.131695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.128869 4309 cpu_manager.go:414] "containerMap: removing container" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" Nov 02 22:22:50.131695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.128895 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" Nov 02 22:22:50.131695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.128944 4309 cpu_manager.go:414] "containerMap: removing container" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" Nov 02 22:22:50.131695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.128952 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" Nov 02 22:22:50.131695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.129020 4309 cpu_manager.go:414] "containerMap: removing container" podUID="65c91168-06a6-4f02-b2b0-48924b4cb525" containerName="revoke-keys" Nov 02 22:22:50.131695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.129029 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c91168-06a6-4f02-b2b0-48924b4cb525" containerName="revoke-keys" Nov 02 22:22:50.131695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.129422 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c91168-06a6-4f02-b2b0-48924b4cb525" containerName="revoke-keys" Nov 02 22:22:50.131695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.129453 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" containerName="main" Nov 02 22:22:50.131695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.129474 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" containerName="main" Nov 02 22:22:50.155317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.155289 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2"] Nov 02 22:22:50.155505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.155417 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.159463 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.159438 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:22:50.159602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.159457 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:22:50.159602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.159513 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:22:50.160600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.160581 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh"] Nov 02 22:22:50.160749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.160620 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2"] Nov 02 22:22:50.160749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.160719 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.161546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.161522 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:22:50.203704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.202233 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" podUID="1292bd76-2ee4-43b5-b3fe-f77417b25e30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:22:50.206920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.204836 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae16f350-d7fa-44dd-bdff-4493c75156b8" path="/var/lib/kubelet/pods/ae16f350-d7fa-44dd-bdff-4493c75156b8/volumes" Nov 02 22:22:50.206920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.205342 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f3196e-3c3f-4748-8789-bc2147772425" path="/var/lib/kubelet/pods/c7f3196e-3c3f-4748-8789-bc2147772425/volumes" Nov 02 22:22:50.206920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.205797 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf36169-2951-4b4a-a8bc-20b19196adaf" path="/var/lib/kubelet/pods/edf36169-2951-4b4a-a8bc-20b19196adaf/volumes" Nov 02 22:22:50.294189 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.294101 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4703b472-f9qp2\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.294384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.294238 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfjr\" (UniqueName: \"kubernetes.io/projected/20ace8dd-4664-431b-a1db-29ad4031a4b1-kube-api-access-kvfjr\") pod \"maas-api-revoke-sub-4703b472-f9qp2\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.294384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.294287 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s28p\" (UniqueName: \"kubernetes.io/projected/68cb7024-a457-4bb4-8296-180cd1ee4617-kube-api-access-9s28p\") pod \"maas-api-revoke-sub-ccd17032-r84kh\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.294384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.294310 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4703b472-f9qp2\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.294560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.294459 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccd17032-r84kh\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.294636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.294566 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccd17032-r84kh\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.404639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.404525 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4703b472-f9qp2\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.405094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.405053 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfjr\" (UniqueName: \"kubernetes.io/projected/20ace8dd-4664-431b-a1db-29ad4031a4b1-kube-api-access-kvfjr\") pod \"maas-api-revoke-sub-4703b472-f9qp2\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.405330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.405267 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s28p\" (UniqueName: \"kubernetes.io/projected/68cb7024-a457-4bb4-8296-180cd1ee4617-kube-api-access-9s28p\") pod \"maas-api-revoke-sub-ccd17032-r84kh\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.405615 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.405305 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4703b472-f9qp2\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.405854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.405819 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccd17032-r84kh\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.406203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.406120 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccd17032-r84kh\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.408144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.408118 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4703b472-f9qp2\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.409550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.409516 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccd17032-r84kh\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.448706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.446967 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s28p\" (UniqueName: \"kubernetes.io/projected/68cb7024-a457-4bb4-8296-180cd1ee4617-kube-api-access-9s28p\") pod \"maas-api-revoke-sub-ccd17032-r84kh\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.449395 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.449335 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4703b472-f9qp2\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.476608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.467149 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccd17032-r84kh\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.476608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.467563 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfjr\" (UniqueName: \"kubernetes.io/projected/20ace8dd-4664-431b-a1db-29ad4031a4b1-kube-api-access-kvfjr\") pod \"maas-api-revoke-sub-4703b472-f9qp2\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.477158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.477131 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:50.504042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.503974 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:50.816955 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:22:50.816912 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cb7024_a457_4bb4_8296_180cd1ee4617.slice/crio-737e78b61fa8d857e4695a4003aa6fad1b203ff0b8c344c3fbd763fcf21caed7 WatchSource:0}: Error finding container 737e78b61fa8d857e4695a4003aa6fad1b203ff0b8c344c3fbd763fcf21caed7: Status 404 returned error can't find the container with id 737e78b61fa8d857e4695a4003aa6fad1b203ff0b8c344c3fbd763fcf21caed7 Nov 02 22:22:50.818545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.818514 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh"] Nov 02 22:22:50.834760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:50.834733 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2"] Nov 02 22:22:50.837014 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:22:50.836989 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ace8dd_4664_431b_a1db_29ad4031a4b1.slice/crio-2d5e3344d001eeec45189a0f659d6c0a1ea90e3fb2879ac826f1c73bbb47ff13 WatchSource:0}: Error finding container 2d5e3344d001eeec45189a0f659d6c0a1ea90e3fb2879ac826f1c73bbb47ff13: Status 404 returned error can't find the container with id 2d5e3344d001eeec45189a0f659d6c0a1ea90e3fb2879ac826f1c73bbb47ff13 Nov 02 22:22:51.744787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.743229 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerID="c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742" exitCode=0 Nov 02 22:22:51.744787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.743507 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" event={"ID":"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54","Type":"ContainerDied","Data":"c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742"} Nov 02 22:22:51.756112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.754253 4309 scope.go:122] "RemoveContainer" containerID="c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742" Nov 02 22:22:51.761228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.759630 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:22:51.761228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.759754 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:22:51.761228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.759813 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:22:51.765350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.763510 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" event={"ID":"68cb7024-a457-4bb4-8296-180cd1ee4617","Type":"ContainerStarted","Data":"9db8920a7d60a1cf2e90e437b27a222977f2cac111630bd87267232963c0fa14"} Nov 02 22:22:51.765350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.763568 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" event={"ID":"68cb7024-a457-4bb4-8296-180cd1ee4617","Type":"ContainerStarted","Data":"737e78b61fa8d857e4695a4003aa6fad1b203ff0b8c344c3fbd763fcf21caed7"} Nov 02 22:22:51.770399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.768743 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" event={"ID":"20ace8dd-4664-431b-a1db-29ad4031a4b1","Type":"ContainerStarted","Data":"b4b0e479a6311a252e928af8c0c94d675cbf1144079b23c3eea0786db457e9b4"} Nov 02 22:22:51.770399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.768798 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" event={"ID":"20ace8dd-4664-431b-a1db-29ad4031a4b1","Type":"ContainerStarted","Data":"2d5e3344d001eeec45189a0f659d6c0a1ea90e3fb2879ac826f1c73bbb47ff13"} Nov 02 22:22:51.824757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.821448 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" podStartSLOduration=1.821430979 podStartE2EDuration="1.821430979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:51.81628378 +0000 UTC m=+1499.754396998" watchObservedRunningTime="2026-11-02 22:22:51.821430979 +0000 UTC m=+1499.759544200" Nov 02 22:22:51.836780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:51.835996 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" podStartSLOduration=1.8359766290000001 podStartE2EDuration="1.835976629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:22:51.833810269 +0000 UTC m=+1499.771923488" watchObservedRunningTime="2026-11-02 22:22:51.835976629 +0000 UTC m=+1499.774089848" Nov 02 22:22:52.619744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:52.619709 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:22:52.779212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:52.779175 4309 generic.go:356] "Generic (PLEG): container finished" podID="68cb7024-a457-4bb4-8296-180cd1ee4617" containerID="9db8920a7d60a1cf2e90e437b27a222977f2cac111630bd87267232963c0fa14" exitCode=0 Nov 02 22:22:52.779667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:52.779258 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" event={"ID":"68cb7024-a457-4bb4-8296-180cd1ee4617","Type":"ContainerDied","Data":"9db8920a7d60a1cf2e90e437b27a222977f2cac111630bd87267232963c0fa14"} Nov 02 22:22:52.780950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:52.780927 4309 generic.go:356] "Generic (PLEG): container finished" podID="20ace8dd-4664-431b-a1db-29ad4031a4b1" containerID="b4b0e479a6311a252e928af8c0c94d675cbf1144079b23c3eea0786db457e9b4" exitCode=0 Nov 02 22:22:52.781064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:52.781006 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" event={"ID":"20ace8dd-4664-431b-a1db-29ad4031a4b1","Type":"ContainerDied","Data":"b4b0e479a6311a252e928af8c0c94d675cbf1144079b23c3eea0786db457e9b4"} Nov 02 22:22:52.798702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:52.792642 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" event={"ID":"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54","Type":"ContainerStarted","Data":"398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4"} Nov 02 22:22:53.371003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:53.370974 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:22:53.371741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:53.371718 4309 patch_prober.go:29] interesting pod/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:22:53.371813 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:53.371767 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 02 22:22:54.093937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.093859 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:54.162355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.162325 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-openshift-ca\") pod \"68cb7024-a457-4bb4-8296-180cd1ee4617\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " Nov 02 22:22:54.162653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.162611 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-openshift-ca" pod "68cb7024-a457-4bb4-8296-180cd1ee4617" (UID: "68cb7024-a457-4bb4-8296-180cd1ee4617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:54.163455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.163435 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cb7024-a457-4bb4-8296-180cd1ee4617-kube-api-access-9s28p\" (UniqueName: \"kubernetes.io/projected/68cb7024-a457-4bb4-8296-180cd1ee4617-kube-api-access-9s28p\") pod \"68cb7024-a457-4bb4-8296-180cd1ee4617\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " Nov 02 22:22:54.163552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.163500 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-xks-ca\") pod \"68cb7024-a457-4bb4-8296-180cd1ee4617\" (UID: \"68cb7024-a457-4bb4-8296-180cd1ee4617\") " Nov 02 22:22:54.164402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.164375 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.167094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.167072 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cb7024-a457-4bb4-8296-180cd1ee4617-kube-api-access-9s28p" pod "68cb7024-a457-4bb4-8296-180cd1ee4617" (UID: "68cb7024-a457-4bb4-8296-180cd1ee4617"). InnerVolumeSpecName "kube-api-access-9s28p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:54.168526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.168465 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-xks-ca" pod "68cb7024-a457-4bb4-8296-180cd1ee4617" (UID: "68cb7024-a457-4bb4-8296-180cd1ee4617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:54.231699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.231625 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:54.272812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.266375 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s28p\" (UniqueName: \"kubernetes.io/projected/68cb7024-a457-4bb4-8296-180cd1ee4617-kube-api-access-9s28p\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.272812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.266415 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68cb7024-a457-4bb4-8296-180cd1ee4617-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.384757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.375484 4309 patch_prober.go:29] interesting pod/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:22:54.384757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.375545 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 02 22:22:54.425734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.423753 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ace8dd-4664-431b-a1db-29ad4031a4b1-kube-api-access-kvfjr\" (UniqueName: \"kubernetes.io/projected/20ace8dd-4664-431b-a1db-29ad4031a4b1-kube-api-access-kvfjr\") pod \"20ace8dd-4664-431b-a1db-29ad4031a4b1\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " Nov 02 22:22:54.425734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.423945 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-openshift-ca\") pod \"20ace8dd-4664-431b-a1db-29ad4031a4b1\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " Nov 02 22:22:54.425734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.424061 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-xks-ca\") pod \"20ace8dd-4664-431b-a1db-29ad4031a4b1\" (UID: \"20ace8dd-4664-431b-a1db-29ad4031a4b1\") " Nov 02 22:22:54.425734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.425106 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-openshift-ca" pod "20ace8dd-4664-431b-a1db-29ad4031a4b1" (UID: "20ace8dd-4664-431b-a1db-29ad4031a4b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:22:54.428644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.428613 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-xks-ca" pod "20ace8dd-4664-431b-a1db-29ad4031a4b1" (UID: "20ace8dd-4664-431b-a1db-29ad4031a4b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:22:54.428824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.428664 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ace8dd-4664-431b-a1db-29ad4031a4b1-kube-api-access-kvfjr" pod "20ace8dd-4664-431b-a1db-29ad4031a4b1" (UID: "20ace8dd-4664-431b-a1db-29ad4031a4b1"). InnerVolumeSpecName "kube-api-access-kvfjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:22:54.527303 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.527268 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.527303 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.527299 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfjr\" (UniqueName: \"kubernetes.io/projected/20ace8dd-4664-431b-a1db-29ad4031a4b1-kube-api-access-kvfjr\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.527303 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.527310 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ace8dd-4664-431b-a1db-29ad4031a4b1-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:22:54.839083 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.839059 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" Nov 02 22:22:54.839354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.839074 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh" event={"ID":"68cb7024-a457-4bb4-8296-180cd1ee4617","Type":"ContainerDied","Data":"737e78b61fa8d857e4695a4003aa6fad1b203ff0b8c344c3fbd763fcf21caed7"} Nov 02 22:22:54.839427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.839376 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="737e78b61fa8d857e4695a4003aa6fad1b203ff0b8c344c3fbd763fcf21caed7" Nov 02 22:22:54.841619 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.841584 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" event={"ID":"20ace8dd-4664-431b-a1db-29ad4031a4b1","Type":"ContainerDied","Data":"2d5e3344d001eeec45189a0f659d6c0a1ea90e3fb2879ac826f1c73bbb47ff13"} Nov 02 22:22:54.841729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.841633 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d5e3344d001eeec45189a0f659d6c0a1ea90e3fb2879ac826f1c73bbb47ff13" Nov 02 22:22:54.841922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:54.841903 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2" Nov 02 22:22:55.371124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:55.371092 4309 patch_prober.go:29] interesting pod/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:22:55.371536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:55.371153 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 02 22:22:56.371194 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:56.371158 4309 patch_prober.go:29] interesting pod/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:22:56.371715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:56.371217 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 02 22:22:56.458258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:56.458228 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:22:56.465722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:56.465700 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:22:57.371279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:57.371243 4309 patch_prober.go:29] interesting pod/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:22:57.371706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:57.371298 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 02 22:22:58.371722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:58.371668 4309 patch_prober.go:29] interesting pod/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Nov 02 22:22:58.372116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:58.371748 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Nov 02 22:22:59.376147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:59.376117 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:22:59.376592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:59.376565 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:22:59.377198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:59.377177 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:22:59.505899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:59.505863 4309 scope.go:122] "RemoveContainer" containerID="a5c11ab4844a4c706c6f0e303692fa420c161f3a2b2a80e8828bbed245f44aad" Nov 02 22:22:59.517907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:59.517884 4309 scope.go:122] "RemoveContainer" containerID="da731014de462a2387db1a9f00aa54648bdd85e97a6bcfb788d795e67c741db3" Nov 02 22:22:59.571899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:59.571863 4309 scope.go:122] "RemoveContainer" containerID="93c6afb0c945ec72242f453b278827d7b4fd0467518b2079a005cc44fa993472" Nov 02 22:22:59.633521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:22:59.633491 4309 scope.go:122] "RemoveContainer" containerID="3096a4177c6f60653c6c948f4a5a46fd05bc06a99a50a50047d63b75f290345e" Nov 02 22:23:00.228698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:00.227936 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" podUID="1292bd76-2ee4-43b5-b3fe-f77417b25e30" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:23:03.401556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:03.401519 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:23:03.435488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:03.432819 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1292bd76-2ee4-43b5-b3fe-f77417b25e30-kube-api-access-6ccwf\" (UniqueName: \"kubernetes.io/projected/1292bd76-2ee4-43b5-b3fe-f77417b25e30-kube-api-access-6ccwf\") pod \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\" (UID: \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\") " Nov 02 22:23:03.435488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:03.432882 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1292bd76-2ee4-43b5-b3fe-f77417b25e30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1292bd76-2ee4-43b5-b3fe-f77417b25e30-plugins-config-volume\") pod \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\" (UID: \"1292bd76-2ee4-43b5-b3fe-f77417b25e30\") " Nov 02 22:23:03.435488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:03.434008 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1292bd76-2ee4-43b5-b3fe-f77417b25e30-plugins-config-volume" pod "1292bd76-2ee4-43b5-b3fe-f77417b25e30" (UID: "1292bd76-2ee4-43b5-b3fe-f77417b25e30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:03.438856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:03.436578 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1292bd76-2ee4-43b5-b3fe-f77417b25e30-kube-api-access-6ccwf" pod "1292bd76-2ee4-43b5-b3fe-f77417b25e30" (UID: "1292bd76-2ee4-43b5-b3fe-f77417b25e30"). InnerVolumeSpecName "kube-api-access-6ccwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:03.535525 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:03.535495 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ccwf\" (UniqueName: \"kubernetes.io/projected/1292bd76-2ee4-43b5-b3fe-f77417b25e30-kube-api-access-6ccwf\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:03.535525 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:03.535523 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1292bd76-2ee4-43b5-b3fe-f77417b25e30-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:04.022753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.022714 4309 generic.go:356] "Generic (PLEG): container finished" podID="1292bd76-2ee4-43b5-b3fe-f77417b25e30" containerID="34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063" exitCode=0 Nov 02 22:23:04.022958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.022804 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" Nov 02 22:23:04.022958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.022836 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" event={"ID":"1292bd76-2ee4-43b5-b3fe-f77417b25e30","Type":"ContainerDied","Data":"34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063"} Nov 02 22:23:04.022958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.022931 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6" event={"ID":"1292bd76-2ee4-43b5-b3fe-f77417b25e30","Type":"ContainerDied","Data":"9cbd22e71b37c16dbbee8e3c0ed5140af8e3f4aa07080d6002148a9014f0f1bd"} Nov 02 22:23:04.023144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.022960 4309 scope.go:122] "RemoveContainer" containerID="34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063" Nov 02 22:23:04.036706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.036688 4309 scope.go:122] "RemoveContainer" containerID="34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063" Nov 02 22:23:04.036998 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:23:04.036971 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063\": container with ID starting with 34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063 not found: ID does not exist" containerID="34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063" Nov 02 22:23:04.037130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.037004 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063"} err="failed to get container status \"34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063\": rpc error: code = NotFound desc = could not find container \"34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063\": container with ID starting with 34c373e041b440a70e630cc71d5d0bd004e3f90b03dc48ddf17e4e6b9765b063 not found: ID does not exist" Nov 02 22:23:04.048702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.048645 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6"] Nov 02 22:23:04.054354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.054323 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c302b6-79c586c55f-qz8w6"] Nov 02 22:23:04.189908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:04.189871 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1292bd76-2ee4-43b5-b3fe-f77417b25e30" path="/var/lib/kubelet/pods/1292bd76-2ee4-43b5-b3fe-f77417b25e30/volumes" Nov 02 22:23:08.111275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:08.111244 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz"] Nov 02 22:23:08.117905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:08.117877 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c302b6-pvmdz"] Nov 02 22:23:08.192121 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:08.192088 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c91168-06a6-4f02-b2b0-48924b4cb525" path="/var/lib/kubelet/pods/65c91168-06a6-4f02-b2b0-48924b4cb525/volumes" Nov 02 22:23:11.597262 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:23:11.597220 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:23:15.488861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:15.488824 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f"] Nov 02 22:23:15.489370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:15.489123 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" containerID="cri-o://1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257" gracePeriod=30 Nov 02 22:23:15.494195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:15.494171 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v"] Nov 02 22:23:15.494597 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:15.494568 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" containerID="cri-o://a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007" gracePeriod=30 Nov 02 22:23:15.505177 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:15.505155 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx"] Nov 02 22:23:15.505628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:15.505568 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" containerID="cri-o://f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914" gracePeriod=30 Nov 02 22:23:20.932307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:20.932267 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:23:20.934649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:20.934609 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:23:21.007001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.006967 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-envoy\") pod \"83f302bd-e66a-43b9-b818-8748b284d98f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " Nov 02 22:23:21.007205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007016 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f302bd-e66a-43b9-b818-8748b284d98f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f302bd-e66a-43b9-b818-8748b284d98f-istiod-ca-cert\") pod \"83f302bd-e66a-43b9-b818-8748b284d98f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " Nov 02 22:23:21.007205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007049 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-kube-api-access-5qpf9\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-kube-api-access-5qpf9\") pod \"c47c857e-7bc5-481a-8296-3d393ba94437\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " Nov 02 22:23:21.007205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007080 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-kube-api-access-4hhh2\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-kube-api-access-4hhh2\") pod \"83f302bd-e66a-43b9-b818-8748b284d98f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " Nov 02 22:23:21.007205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007181 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-envoy\") pod \"c47c857e-7bc5-481a-8296-3d393ba94437\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " Nov 02 22:23:21.007428 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007221 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c47c857e-7bc5-481a-8296-3d393ba94437-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c857e-7bc5-481a-8296-3d393ba94437-istiod-ca-cert\") pod \"c47c857e-7bc5-481a-8296-3d393ba94437\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " Nov 02 22:23:21.007428 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007291 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-credential-socket\") pod \"c47c857e-7bc5-481a-8296-3d393ba94437\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " Nov 02 22:23:21.007428 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007350 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-istio-token\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-istio-token\") pod \"c47c857e-7bc5-481a-8296-3d393ba94437\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " Nov 02 22:23:21.007428 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007411 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c47c857e-7bc5-481a-8296-3d393ba94437-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c857e-7bc5-481a-8296-3d393ba94437-istio-podinfo\") pod \"c47c857e-7bc5-481a-8296-3d393ba94437\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " Nov 02 22:23:21.007638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007505 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f302bd-e66a-43b9-b818-8748b284d98f-istiod-ca-cert" pod "83f302bd-e66a-43b9-b818-8748b284d98f" (UID: "83f302bd-e66a-43b9-b818-8748b284d98f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:21.007815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007795 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83f302bd-e66a-43b9-b818-8748b284d98f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f302bd-e66a-43b9-b818-8748b284d98f-istio-podinfo\") pod \"83f302bd-e66a-43b9-b818-8748b284d98f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " Nov 02 22:23:21.008003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007985 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-data\") pod \"83f302bd-e66a-43b9-b818-8748b284d98f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " Nov 02 22:23:21.008086 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.008019 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-data\") pod \"c47c857e-7bc5-481a-8296-3d393ba94437\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " Nov 02 22:23:21.008150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.007872 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c47c857e-7bc5-481a-8296-3d393ba94437-istiod-ca-cert" pod "c47c857e-7bc5-481a-8296-3d393ba94437" (UID: "c47c857e-7bc5-481a-8296-3d393ba94437"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:21.008150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.008106 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-certs\") pod \"c47c857e-7bc5-481a-8296-3d393ba94437\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " Nov 02 22:23:21.008150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.008144 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-credential-socket\") pod \"83f302bd-e66a-43b9-b818-8748b284d98f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " Nov 02 22:23:21.008302 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.008194 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-socket\") pod \"c47c857e-7bc5-481a-8296-3d393ba94437\" (UID: \"c47c857e-7bc5-481a-8296-3d393ba94437\") " Nov 02 22:23:21.008375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.008362 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-certs\") pod \"83f302bd-e66a-43b9-b818-8748b284d98f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " Nov 02 22:23:21.008450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.008437 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-istio-token\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-istio-token\") pod \"83f302bd-e66a-43b9-b818-8748b284d98f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " Nov 02 22:23:21.008531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.008518 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-socket\") pod \"83f302bd-e66a-43b9-b818-8748b284d98f\" (UID: \"83f302bd-e66a-43b9-b818-8748b284d98f\") " Nov 02 22:23:21.009233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.009212 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83f302bd-e66a-43b9-b818-8748b284d98f-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.009339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.009236 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c47c857e-7bc5-481a-8296-3d393ba94437-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.010562 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.008086 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-credential-socket" pod "c47c857e-7bc5-481a-8296-3d393ba94437" (UID: "c47c857e-7bc5-481a-8296-3d393ba94437"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.010722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.009434 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-socket" pod "83f302bd-e66a-43b9-b818-8748b284d98f" (UID: "83f302bd-e66a-43b9-b818-8748b284d98f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.010823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.009601 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-data" pod "83f302bd-e66a-43b9-b818-8748b284d98f" (UID: "83f302bd-e66a-43b9-b818-8748b284d98f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.010909 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.009801 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-data" pod "c47c857e-7bc5-481a-8296-3d393ba94437" (UID: "c47c857e-7bc5-481a-8296-3d393ba94437"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.010985 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.009971 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-certs" pod "c47c857e-7bc5-481a-8296-3d393ba94437" (UID: "c47c857e-7bc5-481a-8296-3d393ba94437"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.011305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.011284 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-credential-socket" pod "83f302bd-e66a-43b9-b818-8748b284d98f" (UID: "83f302bd-e66a-43b9-b818-8748b284d98f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.011621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.011597 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-socket" pod "c47c857e-7bc5-481a-8296-3d393ba94437" (UID: "c47c857e-7bc5-481a-8296-3d393ba94437"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.011806 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.011788 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-certs" pod "83f302bd-e66a-43b9-b818-8748b284d98f" (UID: "83f302bd-e66a-43b9-b818-8748b284d98f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.012593 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.012556 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c47c857e-7bc5-481a-8296-3d393ba94437-istio-podinfo" pod "c47c857e-7bc5-481a-8296-3d393ba94437" (UID: "c47c857e-7bc5-481a-8296-3d393ba94437"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:23:21.013014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.012978 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-kube-api-access-4hhh2" pod "83f302bd-e66a-43b9-b818-8748b284d98f" (UID: "83f302bd-e66a-43b9-b818-8748b284d98f"). InnerVolumeSpecName "kube-api-access-4hhh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:21.013256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.013003 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83f302bd-e66a-43b9-b818-8748b284d98f-istio-podinfo" pod "83f302bd-e66a-43b9-b818-8748b284d98f" (UID: "83f302bd-e66a-43b9-b818-8748b284d98f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:23:21.013256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.013200 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-envoy" pod "c47c857e-7bc5-481a-8296-3d393ba94437" (UID: "c47c857e-7bc5-481a-8296-3d393ba94437"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.013629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.013540 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-kube-api-access-5qpf9" pod "c47c857e-7bc5-481a-8296-3d393ba94437" (UID: "c47c857e-7bc5-481a-8296-3d393ba94437"). InnerVolumeSpecName "kube-api-access-5qpf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:21.013629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.013599 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-istio-token" pod "c47c857e-7bc5-481a-8296-3d393ba94437" (UID: "c47c857e-7bc5-481a-8296-3d393ba94437"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:21.014989 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.014969 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-istio-token" pod "83f302bd-e66a-43b9-b818-8748b284d98f" (UID: "83f302bd-e66a-43b9-b818-8748b284d98f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:21.015169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.015083 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-envoy" pod "83f302bd-e66a-43b9-b818-8748b284d98f" (UID: "83f302bd-e66a-43b9-b818-8748b284d98f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111170 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111209 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111224 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111237 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c47c857e-7bc5-481a-8296-3d393ba94437-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111251 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83f302bd-e66a-43b9-b818-8748b284d98f-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111264 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111277 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111290 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111302 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111317 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c47c857e-7bc5-481a-8296-3d393ba94437-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111331 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111344 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111357 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111370 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83f302bd-e66a-43b9-b818-8748b284d98f-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111383 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qpf9\" (UniqueName: \"kubernetes.io/projected/c47c857e-7bc5-481a-8296-3d393ba94437-kube-api-access-5qpf9\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.111465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.111397 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hhh2\" (UniqueName: \"kubernetes.io/projected/83f302bd-e66a-43b9-b818-8748b284d98f-kube-api-access-4hhh2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.284197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.284152 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:23:21.360396 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.360360 4309 generic.go:356] "Generic (PLEG): container finished" podID="c47c857e-7bc5-481a-8296-3d393ba94437" containerID="f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914" exitCode=0 Nov 02 22:23:21.360800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.360774 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" event={"ID":"c47c857e-7bc5-481a-8296-3d393ba94437","Type":"ContainerDied","Data":"f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914"} Nov 02 22:23:21.360983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.360967 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" event={"ID":"c47c857e-7bc5-481a-8296-3d393ba94437","Type":"ContainerDied","Data":"b3bee52b66bfe922b6de2a7d0439ff1ef5df4e728b2e595fd59d7b881f2b8291"} Nov 02 22:23:21.361098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.361087 4309 scope.go:122] "RemoveContainer" containerID="f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914" Nov 02 22:23:21.361322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.361308 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx" Nov 02 22:23:21.372756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.370063 4309 generic.go:356] "Generic (PLEG): container finished" podID="83f302bd-e66a-43b9-b818-8748b284d98f" containerID="1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257" exitCode=0 Nov 02 22:23:21.372756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.370243 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" event={"ID":"83f302bd-e66a-43b9-b818-8748b284d98f","Type":"ContainerDied","Data":"1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257"} Nov 02 22:23:21.372756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.370287 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" event={"ID":"83f302bd-e66a-43b9-b818-8748b284d98f","Type":"ContainerDied","Data":"39a4623af600b77711feda88daf2e10b3904a62fa777bddbc813bdb1cfa28f7a"} Nov 02 22:23:21.372756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.370362 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f" Nov 02 22:23:21.428640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.428611 4309 generic.go:356] "Generic (PLEG): container finished" podID="bdbeab10-9763-452d-93ca-a83d99de3652" containerID="a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007" exitCode=0 Nov 02 22:23:21.428886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.428868 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" event={"ID":"bdbeab10-9763-452d-93ca-a83d99de3652","Type":"ContainerDied","Data":"a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007"} Nov 02 22:23:21.429056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.429018 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" event={"ID":"bdbeab10-9763-452d-93ca-a83d99de3652","Type":"ContainerDied","Data":"332fda911c1cf38bd65e97e77e1a69356cb255421dbaf8124afeca9fc5e5213d"} Nov 02 22:23:21.429222 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.429207 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v" Nov 02 22:23:21.430854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.430830 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbeab10-9763-452d-93ca-a83d99de3652-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbeab10-9763-452d-93ca-a83d99de3652-istiod-ca-cert\") pod \"bdbeab10-9763-452d-93ca-a83d99de3652\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " Nov 02 22:23:21.431028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.431013 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-kube-api-access-h24qj\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-kube-api-access-h24qj\") pod \"bdbeab10-9763-452d-93ca-a83d99de3652\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " Nov 02 22:23:21.431199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.431183 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-istio-token\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-istio-token\") pod \"bdbeab10-9763-452d-93ca-a83d99de3652\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " Nov 02 22:23:21.431313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.431301 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-socket\") pod \"bdbeab10-9763-452d-93ca-a83d99de3652\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " Nov 02 22:23:21.431517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.431502 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-credential-socket\") pod \"bdbeab10-9763-452d-93ca-a83d99de3652\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " Nov 02 22:23:21.431632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.431620 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-data\") pod \"bdbeab10-9763-452d-93ca-a83d99de3652\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " Nov 02 22:23:21.431788 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.431775 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-envoy\") pod \"bdbeab10-9763-452d-93ca-a83d99de3652\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " Nov 02 22:23:21.432417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.432000 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-certs\") pod \"bdbeab10-9763-452d-93ca-a83d99de3652\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " Nov 02 22:23:21.432618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.432600 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdbeab10-9763-452d-93ca-a83d99de3652-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbeab10-9763-452d-93ca-a83d99de3652-istio-podinfo\") pod \"bdbeab10-9763-452d-93ca-a83d99de3652\" (UID: \"bdbeab10-9763-452d-93ca-a83d99de3652\") " Nov 02 22:23:21.434163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.434142 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-socket" pod "bdbeab10-9763-452d-93ca-a83d99de3652" (UID: "bdbeab10-9763-452d-93ca-a83d99de3652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.434550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.434533 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbeab10-9763-452d-93ca-a83d99de3652-istiod-ca-cert" pod "bdbeab10-9763-452d-93ca-a83d99de3652" (UID: "bdbeab10-9763-452d-93ca-a83d99de3652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:23:21.443922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.439491 4309 scope.go:122] "RemoveContainer" containerID="f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914" Nov 02 22:23:21.443922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.439993 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-data" pod "bdbeab10-9763-452d-93ca-a83d99de3652" (UID: "bdbeab10-9763-452d-93ca-a83d99de3652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.444460 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:23:21.444123 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914\": container with ID starting with f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914 not found: ID does not exist" containerID="f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914" Nov 02 22:23:21.444460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.444173 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914"} err="failed to get container status \"f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914\": rpc error: code = NotFound desc = could not find container \"f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914\": container with ID starting with f8abdff38ad6293ec47cb2f2b487c568a37a28488b1dfaecd772598c504ef914 not found: ID does not exist" Nov 02 22:23:21.444460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.444203 4309 scope.go:122] "RemoveContainer" containerID="1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257" Nov 02 22:23:21.445632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.444997 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-certs" pod "bdbeab10-9763-452d-93ca-a83d99de3652" (UID: "bdbeab10-9763-452d-93ca-a83d99de3652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.446249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.446229 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdbeab10-9763-452d-93ca-a83d99de3652-istio-podinfo" pod "bdbeab10-9763-452d-93ca-a83d99de3652" (UID: "bdbeab10-9763-452d-93ca-a83d99de3652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:23:21.448032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.447986 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f"] Nov 02 22:23:21.456069 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.456039 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-credential-socket" pod "bdbeab10-9763-452d-93ca-a83d99de3652" (UID: "bdbeab10-9763-452d-93ca-a83d99de3652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.456642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.456475 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-79j7f"] Nov 02 22:23:21.457090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.457059 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-istio-token" pod "bdbeab10-9763-452d-93ca-a83d99de3652" (UID: "bdbeab10-9763-452d-93ca-a83d99de3652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:21.457227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.457205 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-kube-api-access-h24qj" pod "bdbeab10-9763-452d-93ca-a83d99de3652" (UID: "bdbeab10-9763-452d-93ca-a83d99de3652"). InnerVolumeSpecName "kube-api-access-h24qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:23:21.458729 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.458644 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-envoy" pod "bdbeab10-9763-452d-93ca-a83d99de3652" (UID: "bdbeab10-9763-452d-93ca-a83d99de3652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:23:21.476199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.476181 4309 scope.go:122] "RemoveContainer" containerID="1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c" Nov 02 22:23:21.490582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.490548 4309 scope.go:122] "RemoveContainer" containerID="1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257" Nov 02 22:23:21.491190 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:23:21.490948 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257\": container with ID starting with 1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257 not found: ID does not exist" containerID="1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257" Nov 02 22:23:21.491190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.490983 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257"} err="failed to get container status \"1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257\": rpc error: code = NotFound desc = could not find container \"1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257\": container with ID starting with 1ecd744572f997c36ca535df9c1ac6581ce96c4268043684d3bf3e95c5a52257 not found: ID does not exist" Nov 02 22:23:21.491190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.491010 4309 scope.go:122] "RemoveContainer" containerID="1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c" Nov 02 22:23:21.491463 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.491412 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx"] Nov 02 22:23:21.492049 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:23:21.491631 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c\": container with ID starting with 1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c not found: ID does not exist" containerID="1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c" Nov 02 22:23:21.492049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.491656 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c"} err="failed to get container status \"1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c\": rpc error: code = NotFound desc = could not find container \"1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c\": container with ID starting with 1597a0128d36246c734ebf0fd2d8c3f0e7eb879260b7be5de2605a9f768c6b6c not found: ID does not exist" Nov 02 22:23:21.492463 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.492214 4309 scope.go:122] "RemoveContainer" containerID="a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007" Nov 02 22:23:21.498818 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.498794 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-ccmjx"] Nov 02 22:23:21.509355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.509333 4309 scope.go:122] "RemoveContainer" containerID="a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007" Nov 02 22:23:21.509906 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:23:21.509810 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007\": container with ID starting with a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007 not found: ID does not exist" containerID="a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007" Nov 02 22:23:21.509906 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.509851 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007"} err="failed to get container status \"a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007\": rpc error: code = NotFound desc = could not find container \"a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007\": container with ID starting with a6291b68cc538f7aff5a28c2e6a99c5afce2e173f72aaaa6e5047a4121b6c007 not found: ID does not exist" Nov 02 22:23:21.534975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.534948 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.535096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.534978 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.535096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.534992 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.535096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.535000 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.535096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.535009 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.535096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.535018 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdbeab10-9763-452d-93ca-a83d99de3652-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.535096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.535032 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdbeab10-9763-452d-93ca-a83d99de3652-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.535096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.535050 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdbeab10-9763-452d-93ca-a83d99de3652-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.535096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.535059 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h24qj\" (UniqueName: \"kubernetes.io/projected/bdbeab10-9763-452d-93ca-a83d99de3652-kube-api-access-h24qj\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:23:21.755744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.755712 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v"] Nov 02 22:23:21.759908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:21.759884 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c302b6-openshift-default-847b94b857-4hw5v"] Nov 02 22:23:22.191905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:22.191872 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" path="/var/lib/kubelet/pods/83f302bd-e66a-43b9-b818-8748b284d98f/volumes" Nov 02 22:23:22.192478 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:22.192458 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" path="/var/lib/kubelet/pods/bdbeab10-9763-452d-93ca-a83d99de3652/volumes" Nov 02 22:23:22.193014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:22.192994 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" path="/var/lib/kubelet/pods/c47c857e-7bc5-481a-8296-3d393ba94437/volumes" Nov 02 22:23:41.601403 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:23:41.601364 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:23:59.725200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:59.725164 4309 scope.go:122] "RemoveContainer" containerID="44f74f21f7972a3ccde6c013af12c788e2c08fa6cf0467ac38e9a3c7f9709ecb" Nov 02 22:23:59.736412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:59.736391 4309 scope.go:122] "RemoveContainer" containerID="519649c0b371d1ac027e1b6e51f0fa340ac46eef4fcc21e1d4a89dee5ade3aaa" Nov 02 22:23:59.746456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:59.746433 4309 scope.go:122] "RemoveContainer" containerID="ef3b55788e63bd91f11ea1bd040940a38e2d6fd23f35493f6b9b82a9324ae4d6" Nov 02 22:23:59.756127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:23:59.756108 4309 scope.go:122] "RemoveContainer" containerID="0a73ee3f9f62f69c4761b19d96f5137c39507b1c6c1fda33b8ad13c3840b4d50" Nov 02 22:24:00.563702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:00.563104 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq"] Nov 02 22:24:00.638274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:00.637238 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn"] Nov 02 22:24:00.638274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:00.637483 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" containerID="cri-o://82f6313d8638ef8c69c98999c0a9f62f7b1171f143851368cca9548d420afc61" gracePeriod=30 Nov 02 22:24:00.894088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:00.893995 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" containerID="cri-o://52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0" gracePeriod=30 Nov 02 22:24:02.243237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:02.243197 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:24:02.243669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:02.243275 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:24:05.987343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:05.987299 4309 generic.go:356] "Generic (PLEG): container finished" podID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerID="82f6313d8638ef8c69c98999c0a9f62f7b1171f143851368cca9548d420afc61" exitCode=0 Nov 02 22:24:05.987850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:05.987506 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" event={"ID":"2cae483b-0cfe-488a-a47b-ca268dd12374","Type":"ContainerDied","Data":"82f6313d8638ef8c69c98999c0a9f62f7b1171f143851368cca9548d420afc61"} Nov 02 22:24:05.991823 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:24:05.991787 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542c3992_5db9_468b_91c1_0adfa7588628.slice/crio-52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:24:06.029666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.029635 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:24:06.134610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.134580 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-podinfo\") pod \"2cae483b-0cfe-488a-a47b-ca268dd12374\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " Nov 02 22:24:06.135623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.135582 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-data" pod "2cae483b-0cfe-488a-a47b-ca268dd12374" (UID: "2cae483b-0cfe-488a-a47b-ca268dd12374"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.135864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.135846 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-data\") pod \"2cae483b-0cfe-488a-a47b-ca268dd12374\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " Nov 02 22:24:06.136116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.136101 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-credential-socket\") pod \"2cae483b-0cfe-488a-a47b-ca268dd12374\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " Nov 02 22:24:06.136543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.136424 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-credential-socket" pod "2cae483b-0cfe-488a-a47b-ca268dd12374" (UID: "2cae483b-0cfe-488a-a47b-ca268dd12374"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.136788 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.136771 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-kube-api-access-rqb7f\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-kube-api-access-rqb7f\") pod \"2cae483b-0cfe-488a-a47b-ca268dd12374\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " Nov 02 22:24:06.137527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.137509 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cae483b-0cfe-488a-a47b-ca268dd12374-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cae483b-0cfe-488a-a47b-ca268dd12374-istiod-ca-cert\") pod \"2cae483b-0cfe-488a-a47b-ca268dd12374\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " Nov 02 22:24:06.138302 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.138018 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cae483b-0cfe-488a-a47b-ca268dd12374-istiod-ca-cert" pod "2cae483b-0cfe-488a-a47b-ca268dd12374" (UID: "2cae483b-0cfe-488a-a47b-ca268dd12374"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:06.138633 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.138613 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-certs\") pod \"2cae483b-0cfe-488a-a47b-ca268dd12374\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " Nov 02 22:24:06.139321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.139297 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-socket\") pod \"2cae483b-0cfe-488a-a47b-ca268dd12374\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " Nov 02 22:24:06.139760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.139067 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-certs" pod "2cae483b-0cfe-488a-a47b-ca268dd12374" (UID: "2cae483b-0cfe-488a-a47b-ca268dd12374"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.140040 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.139129 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-podinfo" pod "2cae483b-0cfe-488a-a47b-ca268dd12374" (UID: "2cae483b-0cfe-488a-a47b-ca268dd12374"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:24:06.140730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.139623 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-socket" pod "2cae483b-0cfe-488a-a47b-ca268dd12374" (UID: "2cae483b-0cfe-488a-a47b-ca268dd12374"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.140730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.139933 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-envoy\") pod \"2cae483b-0cfe-488a-a47b-ca268dd12374\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " Nov 02 22:24:06.140730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.140293 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-token\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-token\") pod \"2cae483b-0cfe-488a-a47b-ca268dd12374\" (UID: \"2cae483b-0cfe-488a-a47b-ca268dd12374\") " Nov 02 22:24:06.141255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.141225 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.141255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.141253 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cae483b-0cfe-488a-a47b-ca268dd12374-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.141414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.141267 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.141414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.141281 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.141414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.141294 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.141414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.141307 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.141747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.141720 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-kube-api-access-rqb7f" pod "2cae483b-0cfe-488a-a47b-ca268dd12374" (UID: "2cae483b-0cfe-488a-a47b-ca268dd12374"). InnerVolumeSpecName "kube-api-access-rqb7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:06.143238 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.143175 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-envoy" pod "2cae483b-0cfe-488a-a47b-ca268dd12374" (UID: "2cae483b-0cfe-488a-a47b-ca268dd12374"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.143533 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.143491 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-token" pod "2cae483b-0cfe-488a-a47b-ca268dd12374" (UID: "2cae483b-0cfe-488a-a47b-ca268dd12374"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:06.227436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.227406 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:24:06.245278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.245169 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqb7f\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-kube-api-access-rqb7f\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.245278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.245211 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.245278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.245225 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cae483b-0cfe-488a-a47b-ca268dd12374-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.346480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.346441 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-istio-token\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-istio-token\") pod \"542c3992-5db9-468b-91c1-0adfa7588628\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " Nov 02 22:24:06.346761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.346631 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-certs\") pod \"542c3992-5db9-468b-91c1-0adfa7588628\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " Nov 02 22:24:06.346761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.346693 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-kube-api-access-8lqdk\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-kube-api-access-8lqdk\") pod \"542c3992-5db9-468b-91c1-0adfa7588628\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " Nov 02 22:24:06.346891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.346845 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/542c3992-5db9-468b-91c1-0adfa7588628-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c3992-5db9-468b-91c1-0adfa7588628-istio-podinfo\") pod \"542c3992-5db9-468b-91c1-0adfa7588628\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " Nov 02 22:24:06.346949 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.346911 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542c3992-5db9-468b-91c1-0adfa7588628-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c3992-5db9-468b-91c1-0adfa7588628-istiod-ca-cert\") pod \"542c3992-5db9-468b-91c1-0adfa7588628\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " Nov 02 22:24:06.347008 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.346954 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-certs" pod "542c3992-5db9-468b-91c1-0adfa7588628" (UID: "542c3992-5db9-468b-91c1-0adfa7588628"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.347008 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.346998 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-socket\") pod \"542c3992-5db9-468b-91c1-0adfa7588628\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " Nov 02 22:24:06.347125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.347026 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-envoy\") pod \"542c3992-5db9-468b-91c1-0adfa7588628\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " Nov 02 22:24:06.347125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.347091 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-credential-socket\") pod \"542c3992-5db9-468b-91c1-0adfa7588628\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " Nov 02 22:24:06.347239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.347131 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-data\") pod \"542c3992-5db9-468b-91c1-0adfa7588628\" (UID: \"542c3992-5db9-468b-91c1-0adfa7588628\") " Nov 02 22:24:06.347296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.347249 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-socket" pod "542c3992-5db9-468b-91c1-0adfa7588628" (UID: "542c3992-5db9-468b-91c1-0adfa7588628"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.347357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.347293 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542c3992-5db9-468b-91c1-0adfa7588628-istiod-ca-cert" pod "542c3992-5db9-468b-91c1-0adfa7588628" (UID: "542c3992-5db9-468b-91c1-0adfa7588628"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:06.347561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.347537 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-credential-socket" pod "542c3992-5db9-468b-91c1-0adfa7588628" (UID: "542c3992-5db9-468b-91c1-0adfa7588628"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.347776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.347753 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-data" pod "542c3992-5db9-468b-91c1-0adfa7588628" (UID: "542c3992-5db9-468b-91c1-0adfa7588628"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.348018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.347997 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/542c3992-5db9-468b-91c1-0adfa7588628-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.348114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.348024 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.348114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.348039 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.348114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.348054 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.349170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.349146 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-kube-api-access-8lqdk" pod "542c3992-5db9-468b-91c1-0adfa7588628" (UID: "542c3992-5db9-468b-91c1-0adfa7588628"). InnerVolumeSpecName "kube-api-access-8lqdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:06.349290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.349266 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-istio-token" pod "542c3992-5db9-468b-91c1-0adfa7588628" (UID: "542c3992-5db9-468b-91c1-0adfa7588628"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:06.349380 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.349363 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-envoy" pod "542c3992-5db9-468b-91c1-0adfa7588628" (UID: "542c3992-5db9-468b-91c1-0adfa7588628"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:24:06.349429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.349386 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/542c3992-5db9-468b-91c1-0adfa7588628-istio-podinfo" pod "542c3992-5db9-468b-91c1-0adfa7588628" (UID: "542c3992-5db9-468b-91c1-0adfa7588628"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:24:06.454464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.454338 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.454464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.454379 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lqdk\" (UniqueName: \"kubernetes.io/projected/542c3992-5db9-468b-91c1-0adfa7588628-kube-api-access-8lqdk\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.454464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.454396 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/542c3992-5db9-468b-91c1-0adfa7588628-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.454464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.454409 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.454464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.454425 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/542c3992-5db9-468b-91c1-0adfa7588628-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:06.997895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.997800 4309 generic.go:356] "Generic (PLEG): container finished" podID="542c3992-5db9-468b-91c1-0adfa7588628" containerID="52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0" exitCode=0 Nov 02 22:24:06.997895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.997864 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" Nov 02 22:24:06.997895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.997882 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" event={"ID":"542c3992-5db9-468b-91c1-0adfa7588628","Type":"ContainerDied","Data":"52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0"} Nov 02 22:24:06.998397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.997949 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq" event={"ID":"542c3992-5db9-468b-91c1-0adfa7588628","Type":"ContainerDied","Data":"18bfd9b65ee6d6d16753935cae9520423e85d32b02bfe485feda552caab60626"} Nov 02 22:24:06.998397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:06.997982 4309 scope.go:122] "RemoveContainer" containerID="52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0" Nov 02 22:24:07.000296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:07.000274 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" event={"ID":"2cae483b-0cfe-488a-a47b-ca268dd12374","Type":"ContainerDied","Data":"f8aedd0fe16c22b834b08b0a9d36409582dc655c3c7471a832d44ce80ed31810"} Nov 02 22:24:07.000411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:07.000296 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn" Nov 02 22:24:07.010897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:07.010880 4309 scope.go:122] "RemoveContainer" containerID="52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0" Nov 02 22:24:07.011189 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:24:07.011170 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0\": container with ID starting with 52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0 not found: ID does not exist" containerID="52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0" Nov 02 22:24:07.011270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:07.011200 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0"} err="failed to get container status \"52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0\": rpc error: code = NotFound desc = could not find container \"52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0\": container with ID starting with 52eddd247fdf5cc8dc46b2a19ed0480d25627ba79d6d70d1a2993432f0dc35c0 not found: ID does not exist" Nov 02 22:24:07.011270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:07.011217 4309 scope.go:122] "RemoveContainer" containerID="82f6313d8638ef8c69c98999c0a9f62f7b1171f143851368cca9548d420afc61" Nov 02 22:24:07.054486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:07.054451 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn"] Nov 02 22:24:07.065419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:07.065390 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-z9qsn"] Nov 02 22:24:07.079899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:07.079871 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq"] Nov 02 22:24:07.084501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:07.084474 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-vjkgq"] Nov 02 22:24:08.191795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:08.191758 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" path="/var/lib/kubelet/pods/2cae483b-0cfe-488a-a47b-ca268dd12374/volumes" Nov 02 22:24:08.192344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:08.192328 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542c3992-5db9-468b-91c1-0adfa7588628" path="/var/lib/kubelet/pods/542c3992-5db9-468b-91c1-0adfa7588628/volumes" Nov 02 22:24:11.597760 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:24:11.597720 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:24:15.917089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.917049 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll"] Nov 02 22:24:15.918195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918174 4309 cpu_manager.go:414] "containerMap: removing container" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" Nov 02 22:24:15.918264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918198 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" Nov 02 22:24:15.918264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918241 4309 cpu_manager.go:414] "containerMap: removing container" podUID="68cb7024-a457-4bb4-8296-180cd1ee4617" containerName="revoke-keys" Nov 02 22:24:15.918264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918249 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cb7024-a457-4bb4-8296-180cd1ee4617" containerName="revoke-keys" Nov 02 22:24:15.918356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918270 4309 cpu_manager.go:414] "containerMap: removing container" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" Nov 02 22:24:15.918356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918278 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" Nov 02 22:24:15.918356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918291 4309 cpu_manager.go:414] "containerMap: removing container" podUID="20ace8dd-4664-431b-a1db-29ad4031a4b1" containerName="revoke-keys" Nov 02 22:24:15.918356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918297 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ace8dd-4664-431b-a1db-29ad4031a4b1" containerName="revoke-keys" Nov 02 22:24:15.918356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918325 4309 cpu_manager.go:414] "containerMap: removing container" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" Nov 02 22:24:15.918356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918332 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" Nov 02 22:24:15.918356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918342 4309 cpu_manager.go:414] "containerMap: removing container" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" Nov 02 22:24:15.918356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918348 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" Nov 02 22:24:15.918600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918363 4309 cpu_manager.go:414] "containerMap: removing container" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" Nov 02 22:24:15.918600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918370 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" Nov 02 22:24:15.918600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918402 4309 cpu_manager.go:414] "containerMap: removing container" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" Nov 02 22:24:15.918600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918409 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" Nov 02 22:24:15.918600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918425 4309 cpu_manager.go:414] "containerMap: removing container" podUID="1292bd76-2ee4-43b5-b3fe-f77417b25e30" containerName="payload-processing" Nov 02 22:24:15.918600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918433 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1292bd76-2ee4-43b5-b3fe-f77417b25e30" containerName="payload-processing" Nov 02 22:24:15.918600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918584 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbeab10-9763-452d-93ca-a83d99de3652" containerName="istio-proxy" Nov 02 22:24:15.918600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918594 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" Nov 02 22:24:15.918861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918606 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="542c3992-5db9-468b-91c1-0adfa7588628" containerName="istio-proxy" Nov 02 22:24:15.918861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918620 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f302bd-e66a-43b9-b818-8748b284d98f" containerName="istio-proxy" Nov 02 22:24:15.918861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918631 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="1292bd76-2ee4-43b5-b3fe-f77417b25e30" containerName="payload-processing" Nov 02 22:24:15.918861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918641 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="c47c857e-7bc5-481a-8296-3d393ba94437" containerName="istio-proxy" Nov 02 22:24:15.918861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918653 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cb7024-a457-4bb4-8296-180cd1ee4617" containerName="revoke-keys" Nov 02 22:24:15.918861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918662 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cae483b-0cfe-488a-a47b-ca268dd12374" containerName="istio-proxy" Nov 02 22:24:15.918861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.918689 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ace8dd-4664-431b-a1db-29ad4031a4b1" containerName="revoke-keys" Nov 02 22:24:15.923266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.923239 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:15.929250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.929223 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll"] Nov 02 22:24:15.930198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.930177 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:24:15.930349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.930327 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:24:15.930437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.930362 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:24:15.930437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:15.930390 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:24:16.056097 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.056057 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beac5662-nljll\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.056283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.056195 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpf2\" (UniqueName: \"kubernetes.io/projected/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-kube-api-access-7vpf2\") pod \"maas-api-revoke-sub-beac5662-nljll\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.056330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.056298 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beac5662-nljll\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.163657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.163608 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beac5662-nljll\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.164535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.164508 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpf2\" (UniqueName: \"kubernetes.io/projected/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-kube-api-access-7vpf2\") pod \"maas-api-revoke-sub-beac5662-nljll\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.164859 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.164840 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beac5662-nljll\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.165832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.165802 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beac5662-nljll\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.177716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.177566 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpf2\" (UniqueName: \"kubernetes.io/projected/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-kube-api-access-7vpf2\") pod \"maas-api-revoke-sub-beac5662-nljll\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.218115 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.218083 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beac5662-nljll\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.246992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.246955 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:16.541643 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:24:16.541616 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2bd6a4_de5a_4c4c_9266_0a43061d2528.slice/crio-e0c288fe75ba8803f25fea296815e061b9857ced1b5557165a93df1fb3ca1763 WatchSource:0}: Error finding container e0c288fe75ba8803f25fea296815e061b9857ced1b5557165a93df1fb3ca1763: Status 404 returned error can't find the container with id e0c288fe75ba8803f25fea296815e061b9857ced1b5557165a93df1fb3ca1763 Nov 02 22:24:16.541769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:16.541732 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll"] Nov 02 22:24:17.088781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:17.088744 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" event={"ID":"4d2bd6a4-de5a-4c4c-9266-0a43061d2528","Type":"ContainerStarted","Data":"c0eb787e009d05007abf806b7a6628d562cbd708b17957198ad914d3283612ce"} Nov 02 22:24:17.089280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:17.088809 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" event={"ID":"4d2bd6a4-de5a-4c4c-9266-0a43061d2528","Type":"ContainerStarted","Data":"e0c288fe75ba8803f25fea296815e061b9857ced1b5557165a93df1fb3ca1763"} Nov 02 22:24:17.106326 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:17.106270 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" podStartSLOduration=2.106255337 podStartE2EDuration="2.106255337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:17.105428017 +0000 UTC m=+1585.043541236" watchObservedRunningTime="2026-11-02 22:24:17.106255337 +0000 UTC m=+1585.044368554" Nov 02 22:24:18.193776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:18.193739 4309 generic.go:356] "Generic (PLEG): container finished" podID="4d2bd6a4-de5a-4c4c-9266-0a43061d2528" containerID="c0eb787e009d05007abf806b7a6628d562cbd708b17957198ad914d3283612ce" exitCode=0 Nov 02 22:24:18.205973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:18.205942 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" event={"ID":"4d2bd6a4-de5a-4c4c-9266-0a43061d2528","Type":"ContainerDied","Data":"c0eb787e009d05007abf806b7a6628d562cbd708b17957198ad914d3283612ce"} Nov 02 22:24:19.380984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.380958 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:19.458484 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.458395 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-openshift-ca\") pod \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " Nov 02 22:24:19.458653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.458578 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-kube-api-access-7vpf2\" (UniqueName: \"kubernetes.io/projected/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-kube-api-access-7vpf2\") pod \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " Nov 02 22:24:19.458792 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.458666 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-xks-ca\") pod \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\" (UID: \"4d2bd6a4-de5a-4c4c-9266-0a43061d2528\") " Nov 02 22:24:19.458903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.458875 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-openshift-ca" pod "4d2bd6a4-de5a-4c4c-9266-0a43061d2528" (UID: "4d2bd6a4-de5a-4c4c-9266-0a43061d2528"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:19.459508 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.459482 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:19.460991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.460969 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-xks-ca" pod "4d2bd6a4-de5a-4c4c-9266-0a43061d2528" (UID: "4d2bd6a4-de5a-4c4c-9266-0a43061d2528"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:24:19.461355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.461328 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-kube-api-access-7vpf2" pod "4d2bd6a4-de5a-4c4c-9266-0a43061d2528" (UID: "4d2bd6a4-de5a-4c4c-9266-0a43061d2528"). InnerVolumeSpecName "kube-api-access-7vpf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:19.583899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.571844 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vpf2\" (UniqueName: \"kubernetes.io/projected/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-kube-api-access-7vpf2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:19.583899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:19.571880 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d2bd6a4-de5a-4c4c-9266-0a43061d2528-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:20.239565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:20.239524 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" event={"ID":"4d2bd6a4-de5a-4c4c-9266-0a43061d2528","Type":"ContainerDied","Data":"e0c288fe75ba8803f25fea296815e061b9857ced1b5557165a93df1fb3ca1763"} Nov 02 22:24:20.239804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:20.239588 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c288fe75ba8803f25fea296815e061b9857ced1b5557165a93df1fb3ca1763" Nov 02 22:24:20.239804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:20.239657 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll" Nov 02 22:24:27.129774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.129732 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9"] Nov 02 22:24:27.131807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.131779 4309 cpu_manager.go:414] "containerMap: removing container" podUID="4d2bd6a4-de5a-4c4c-9266-0a43061d2528" containerName="revoke-keys" Nov 02 22:24:27.132017 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.132001 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2bd6a4-de5a-4c4c-9266-0a43061d2528" containerName="revoke-keys" Nov 02 22:24:27.132469 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.132449 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2bd6a4-de5a-4c4c-9266-0a43061d2528" containerName="revoke-keys" Nov 02 22:24:27.170649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.170620 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.187961 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.186641 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:24:27.202181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.187148 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:24:27.202272 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.187212 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:24:27.202324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.193881 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:24:27.202370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.200292 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9"] Nov 02 22:24:27.264278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.264240 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f39394c5-87mp9\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.264456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.264394 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f39394c5-87mp9\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.264572 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.264549 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvj8q\" (UniqueName: \"kubernetes.io/projected/d5961abd-9636-44b4-b650-4a2a65c4b603-kube-api-access-jvj8q\") pod \"maas-api-revoke-sub-f39394c5-87mp9\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.366383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.366346 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f39394c5-87mp9\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.366565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.366419 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f39394c5-87mp9\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.366565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.366486 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvj8q\" (UniqueName: \"kubernetes.io/projected/d5961abd-9636-44b4-b650-4a2a65c4b603-kube-api-access-jvj8q\") pod \"maas-api-revoke-sub-f39394c5-87mp9\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.367157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.367129 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f39394c5-87mp9\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.368822 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.368796 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f39394c5-87mp9\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.373796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.373775 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvj8q\" (UniqueName: \"kubernetes.io/projected/d5961abd-9636-44b4-b650-4a2a65c4b603-kube-api-access-jvj8q\") pod \"maas-api-revoke-sub-f39394c5-87mp9\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.518967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.518882 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:27.867609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:27.849551 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9"] Nov 02 22:24:28.373713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:28.370494 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" event={"ID":"d5961abd-9636-44b4-b650-4a2a65c4b603","Type":"ContainerStarted","Data":"1c0fc02f7e322b70462063aa153006d57ad2bcea32e50b2bc26ec31c175c1b9f"} Nov 02 22:24:29.380669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:29.380636 4309 generic.go:356] "Generic (PLEG): container finished" podID="d5961abd-9636-44b4-b650-4a2a65c4b603" containerID="c54a53de1a7e3d2ed40166ae9b8abdb683aa38a16ba091a452f840131e92b44a" exitCode=0 Nov 02 22:24:29.381085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:29.380704 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" event={"ID":"d5961abd-9636-44b4-b650-4a2a65c4b603","Type":"ContainerDied","Data":"c54a53de1a7e3d2ed40166ae9b8abdb683aa38a16ba091a452f840131e92b44a"} Nov 02 22:24:30.654497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.654146 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:30.737793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.737757 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-openshift-ca\") pod \"d5961abd-9636-44b4-b650-4a2a65c4b603\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " Nov 02 22:24:30.738132 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.737948 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5961abd-9636-44b4-b650-4a2a65c4b603-kube-api-access-jvj8q\" (UniqueName: \"kubernetes.io/projected/d5961abd-9636-44b4-b650-4a2a65c4b603-kube-api-access-jvj8q\") pod \"d5961abd-9636-44b4-b650-4a2a65c4b603\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " Nov 02 22:24:30.738132 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.738109 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-xks-ca\") pod \"d5961abd-9636-44b4-b650-4a2a65c4b603\" (UID: \"d5961abd-9636-44b4-b650-4a2a65c4b603\") " Nov 02 22:24:30.738269 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.738197 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-openshift-ca" pod "d5961abd-9636-44b4-b650-4a2a65c4b603" (UID: "d5961abd-9636-44b4-b650-4a2a65c4b603"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:30.738937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.738912 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:30.740861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.740818 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5961abd-9636-44b4-b650-4a2a65c4b603-kube-api-access-jvj8q" pod "d5961abd-9636-44b4-b650-4a2a65c4b603" (UID: "d5961abd-9636-44b4-b650-4a2a65c4b603"). InnerVolumeSpecName "kube-api-access-jvj8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:30.741282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.741256 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-xks-ca" pod "d5961abd-9636-44b4-b650-4a2a65c4b603" (UID: "d5961abd-9636-44b4-b650-4a2a65c4b603"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:24:30.840448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.840413 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5961abd-9636-44b4-b650-4a2a65c4b603-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:30.840448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:30.840443 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvj8q\" (UniqueName: \"kubernetes.io/projected/d5961abd-9636-44b4-b650-4a2a65c4b603-kube-api-access-jvj8q\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:31.025922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:31.025826 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j"] Nov 02 22:24:31.033118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:31.033082 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f652337-8lc9j"] Nov 02 22:24:31.400842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:31.400810 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" Nov 02 22:24:31.401027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:31.400835 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9" event={"ID":"d5961abd-9636-44b4-b650-4a2a65c4b603","Type":"ContainerDied","Data":"1c0fc02f7e322b70462063aa153006d57ad2bcea32e50b2bc26ec31c175c1b9f"} Nov 02 22:24:31.401027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:31.400892 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c0fc02f7e322b70462063aa153006d57ad2bcea32e50b2bc26ec31c175c1b9f" Nov 02 22:24:32.191175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:32.191135 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db" path="/var/lib/kubelet/pods/d8b5d974-cf70-4c0b-8b0f-86ef31c3c7db/volumes" Nov 02 22:24:38.019750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.019713 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f"] Nov 02 22:24:38.021465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.021437 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d5961abd-9636-44b4-b650-4a2a65c4b603" containerName="revoke-keys" Nov 02 22:24:38.021660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.021646 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5961abd-9636-44b4-b650-4a2a65c4b603" containerName="revoke-keys" Nov 02 22:24:38.024841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.022164 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5961abd-9636-44b4-b650-4a2a65c4b603" containerName="revoke-keys" Nov 02 22:24:38.031546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.030786 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f"] Nov 02 22:24:38.031546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.030892 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.034491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.034465 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:24:38.034603 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.034576 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:24:38.036776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.036476 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:24:38.036776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.036573 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:24:38.156757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.156722 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.156944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.156793 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.157173 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.157141 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc2gq\" (UniqueName: \"kubernetes.io/projected/d2eab28f-7f09-4e24-a291-6141cb5a5560-kube-api-access-qc2gq\") pod \"maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.259704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.259068 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.259704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.259148 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.259704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.259389 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc2gq\" (UniqueName: \"kubernetes.io/projected/d2eab28f-7f09-4e24-a291-6141cb5a5560-kube-api-access-qc2gq\") pod \"maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.260603 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.260571 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.262682 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.262651 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.269038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.267586 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc2gq\" (UniqueName: \"kubernetes.io/projected/d2eab28f-7f09-4e24-a291-6141cb5a5560-kube-api-access-qc2gq\") pod \"maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.360541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.360509 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:38.667145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:38.667105 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f"] Nov 02 22:24:39.487460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:39.487368 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" event={"ID":"d2eab28f-7f09-4e24-a291-6141cb5a5560","Type":"ContainerStarted","Data":"32f7f31c9569236e20444e337b318fd7442aa761ebcb3c86e526a972c5b47b35"} Nov 02 22:24:39.487460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:39.487442 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" event={"ID":"d2eab28f-7f09-4e24-a291-6141cb5a5560","Type":"ContainerStarted","Data":"c95d65b503a21d7453b9fc3825efa2d55412469d9dee1422ea1a27203e87f65b"} Nov 02 22:24:39.507296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:39.507235 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" podStartSLOduration=2.507216661 podStartE2EDuration="2.507216661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:39.505831016 +0000 UTC m=+1607.443944234" watchObservedRunningTime="2026-11-02 22:24:39.507216661 +0000 UTC m=+1607.445329880" Nov 02 22:24:40.508442 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:40.508403 4309 generic.go:356] "Generic (PLEG): container finished" podID="d2eab28f-7f09-4e24-a291-6141cb5a5560" containerID="32f7f31c9569236e20444e337b318fd7442aa761ebcb3c86e526a972c5b47b35" exitCode=0 Nov 02 22:24:40.508991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:40.508538 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" event={"ID":"d2eab28f-7f09-4e24-a291-6141cb5a5560","Type":"ContainerDied","Data":"32f7f31c9569236e20444e337b318fd7442aa761ebcb3c86e526a972c5b47b35"} Nov 02 22:24:40.759465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:40.759381 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql"] Nov 02 22:24:40.765402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:40.765372 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:40.775576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:40.775542 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql"] Nov 02 22:24:40.944520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:40.944475 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/7af63f12-8f7b-4505-ae0f-99e851824cd6-kube-api-access-lfftd\") pod \"maas-api-revoke-sub-37acced6-2gbql\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:40.944786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:40.944557 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37acced6-2gbql\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:40.944786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:40.944754 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37acced6-2gbql\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:41.076699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.072281 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/7af63f12-8f7b-4505-ae0f-99e851824cd6-kube-api-access-lfftd\") pod \"maas-api-revoke-sub-37acced6-2gbql\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:41.076699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.072425 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37acced6-2gbql\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:41.076699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.072581 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37acced6-2gbql\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:41.102700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.098884 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37acced6-2gbql\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:41.123272 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.117154 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37acced6-2gbql\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:41.133530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.131509 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/7af63f12-8f7b-4505-ae0f-99e851824cd6-kube-api-access-lfftd\") pod \"maas-api-revoke-sub-37acced6-2gbql\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:41.421470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.421384 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:41.635924 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:24:41.635878 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:24:41.727296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.727261 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql"] Nov 02 22:24:41.817246 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.817216 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:41.891553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.891525 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-openshift-ca\") pod \"d2eab28f-7f09-4e24-a291-6141cb5a5560\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " Nov 02 22:24:41.891963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.891935 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-openshift-ca" pod "d2eab28f-7f09-4e24-a291-6141cb5a5560" (UID: "d2eab28f-7f09-4e24-a291-6141cb5a5560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:41.892088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.891938 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2eab28f-7f09-4e24-a291-6141cb5a5560-kube-api-access-qc2gq\" (UniqueName: \"kubernetes.io/projected/d2eab28f-7f09-4e24-a291-6141cb5a5560-kube-api-access-qc2gq\") pod \"d2eab28f-7f09-4e24-a291-6141cb5a5560\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " Nov 02 22:24:41.892152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.892122 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-xks-ca\") pod \"d2eab28f-7f09-4e24-a291-6141cb5a5560\" (UID: \"d2eab28f-7f09-4e24-a291-6141cb5a5560\") " Nov 02 22:24:41.892889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.892868 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:41.894156 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.894135 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-xks-ca" pod "d2eab28f-7f09-4e24-a291-6141cb5a5560" (UID: "d2eab28f-7f09-4e24-a291-6141cb5a5560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:24:41.894736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.894714 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2eab28f-7f09-4e24-a291-6141cb5a5560-kube-api-access-qc2gq" pod "d2eab28f-7f09-4e24-a291-6141cb5a5560" (UID: "d2eab28f-7f09-4e24-a291-6141cb5a5560"). InnerVolumeSpecName "kube-api-access-qc2gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:41.995131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.995057 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc2gq\" (UniqueName: \"kubernetes.io/projected/d2eab28f-7f09-4e24-a291-6141cb5a5560-kube-api-access-qc2gq\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:41.995131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:41.995094 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2eab28f-7f09-4e24-a291-6141cb5a5560-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:42.546907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:42.546864 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" event={"ID":"7af63f12-8f7b-4505-ae0f-99e851824cd6","Type":"ContainerStarted","Data":"5af98d36239a5d3a3f7a5042250843b99f28b6460e0c75c04ae8e1246cb1e441"} Nov 02 22:24:42.547112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:42.546967 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" event={"ID":"7af63f12-8f7b-4505-ae0f-99e851824cd6","Type":"ContainerStarted","Data":"67ee8354b185e180fb25e1f4aa99f17bfec3e67239dae4b3f99e69e3f1e921fc"} Nov 02 22:24:42.548343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:42.548319 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" Nov 02 22:24:42.548460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:42.548322 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f" event={"ID":"d2eab28f-7f09-4e24-a291-6141cb5a5560","Type":"ContainerDied","Data":"c95d65b503a21d7453b9fc3825efa2d55412469d9dee1422ea1a27203e87f65b"} Nov 02 22:24:42.548460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:42.548436 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c95d65b503a21d7453b9fc3825efa2d55412469d9dee1422ea1a27203e87f65b" Nov 02 22:24:42.570832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:42.570781 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" podStartSLOduration=2.570764538 podStartE2EDuration="2.570764538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:42.57001287 +0000 UTC m=+1610.508126091" watchObservedRunningTime="2026-11-02 22:24:42.570764538 +0000 UTC m=+1610.508877757" Nov 02 22:24:43.559843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:43.559804 4309 generic.go:356] "Generic (PLEG): container finished" podID="7af63f12-8f7b-4505-ae0f-99e851824cd6" containerID="5af98d36239a5d3a3f7a5042250843b99f28b6460e0c75c04ae8e1246cb1e441" exitCode=0 Nov 02 22:24:43.560264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:43.559881 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" event={"ID":"7af63f12-8f7b-4505-ae0f-99e851824cd6","Type":"ContainerDied","Data":"5af98d36239a5d3a3f7a5042250843b99f28b6460e0c75c04ae8e1246cb1e441"} Nov 02 22:24:44.816700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.811337 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:44.861886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.861806 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-openshift-ca\") pod \"7af63f12-8f7b-4505-ae0f-99e851824cd6\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " Nov 02 22:24:44.862048 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.862031 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-xks-ca\") pod \"7af63f12-8f7b-4505-ae0f-99e851824cd6\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " Nov 02 22:24:44.862103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.862092 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af63f12-8f7b-4505-ae0f-99e851824cd6-kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/7af63f12-8f7b-4505-ae0f-99e851824cd6-kube-api-access-lfftd\") pod \"7af63f12-8f7b-4505-ae0f-99e851824cd6\" (UID: \"7af63f12-8f7b-4505-ae0f-99e851824cd6\") " Nov 02 22:24:44.862288 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.862258 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-openshift-ca" pod "7af63f12-8f7b-4505-ae0f-99e851824cd6" (UID: "7af63f12-8f7b-4505-ae0f-99e851824cd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:44.863117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.863089 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:44.864308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.864279 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-xks-ca" pod "7af63f12-8f7b-4505-ae0f-99e851824cd6" (UID: "7af63f12-8f7b-4505-ae0f-99e851824cd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:24:44.864404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.864285 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af63f12-8f7b-4505-ae0f-99e851824cd6-kube-api-access-lfftd" pod "7af63f12-8f7b-4505-ae0f-99e851824cd6" (UID: "7af63f12-8f7b-4505-ae0f-99e851824cd6"). InnerVolumeSpecName "kube-api-access-lfftd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:44.965162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.965131 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af63f12-8f7b-4505-ae0f-99e851824cd6-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:44.965162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:44.965161 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfftd\" (UniqueName: \"kubernetes.io/projected/7af63f12-8f7b-4505-ae0f-99e851824cd6-kube-api-access-lfftd\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:45.520398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.520365 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll"] Nov 02 22:24:45.521233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.521213 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7af63f12-8f7b-4505-ae0f-99e851824cd6" containerName="revoke-keys" Nov 02 22:24:45.521233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.521233 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af63f12-8f7b-4505-ae0f-99e851824cd6" containerName="revoke-keys" Nov 02 22:24:45.521385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.521272 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d2eab28f-7f09-4e24-a291-6141cb5a5560" containerName="revoke-keys" Nov 02 22:24:45.521385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.521277 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2eab28f-7f09-4e24-a291-6141cb5a5560" containerName="revoke-keys" Nov 02 22:24:45.521452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.521405 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af63f12-8f7b-4505-ae0f-99e851824cd6" containerName="revoke-keys" Nov 02 22:24:45.521452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.521419 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2eab28f-7f09-4e24-a291-6141cb5a5560" containerName="revoke-keys" Nov 02 22:24:45.526727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.526705 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.533152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.533120 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll"] Nov 02 22:24:45.579537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.579501 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" event={"ID":"7af63f12-8f7b-4505-ae0f-99e851824cd6","Type":"ContainerDied","Data":"67ee8354b185e180fb25e1f4aa99f17bfec3e67239dae4b3f99e69e3f1e921fc"} Nov 02 22:24:45.579757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.579560 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67ee8354b185e180fb25e1f4aa99f17bfec3e67239dae4b3f99e69e3f1e921fc" Nov 02 22:24:45.579757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.579626 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql" Nov 02 22:24:45.682184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.682145 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdgw\" (UniqueName: \"kubernetes.io/projected/00258507-e531-4b1f-a3ad-fa55c532c503-kube-api-access-skdgw\") pod \"maas-api-revoke-sub-8b8200ed-blvll\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.682428 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.682402 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b8200ed-blvll\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.682628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.682606 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b8200ed-blvll\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.784195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.784102 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b8200ed-blvll\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.784344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.784253 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdgw\" (UniqueName: \"kubernetes.io/projected/00258507-e531-4b1f-a3ad-fa55c532c503-kube-api-access-skdgw\") pod \"maas-api-revoke-sub-8b8200ed-blvll\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.784481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.784463 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b8200ed-blvll\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.784995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.784959 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b8200ed-blvll\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.786961 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.786931 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b8200ed-blvll\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.792038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.792009 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdgw\" (UniqueName: \"kubernetes.io/projected/00258507-e531-4b1f-a3ad-fa55c532c503-kube-api-access-skdgw\") pod \"maas-api-revoke-sub-8b8200ed-blvll\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:45.847890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:45.847862 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:46.143389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:46.143351 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll"] Nov 02 22:24:46.147248 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:24:46.147194 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00258507_e531_4b1f_a3ad_fa55c532c503.slice/crio-98c4a4d8baeb973028764e08feb1681e172760a17cb61bbcc14945be8fb59198 WatchSource:0}: Error finding container 98c4a4d8baeb973028764e08feb1681e172760a17cb61bbcc14945be8fb59198: Status 404 returned error can't find the container with id 98c4a4d8baeb973028764e08feb1681e172760a17cb61bbcc14945be8fb59198 Nov 02 22:24:46.589994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:46.589956 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" event={"ID":"00258507-e531-4b1f-a3ad-fa55c532c503","Type":"ContainerStarted","Data":"0811a8941f0fe9a56123d51ba59bdc97071dd05e5eced04b2d4052f110bafc3d"} Nov 02 22:24:46.590180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:46.590024 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" event={"ID":"00258507-e531-4b1f-a3ad-fa55c532c503","Type":"ContainerStarted","Data":"98c4a4d8baeb973028764e08feb1681e172760a17cb61bbcc14945be8fb59198"} Nov 02 22:24:46.613046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:46.612991 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" podStartSLOduration=1.612973936 podStartE2EDuration="1.612973936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:46.604505996 +0000 UTC m=+1614.542619214" watchObservedRunningTime="2026-11-02 22:24:46.612973936 +0000 UTC m=+1614.551087155" Nov 02 22:24:47.604401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:47.604365 4309 generic.go:356] "Generic (PLEG): container finished" podID="00258507-e531-4b1f-a3ad-fa55c532c503" containerID="0811a8941f0fe9a56123d51ba59bdc97071dd05e5eced04b2d4052f110bafc3d" exitCode=0 Nov 02 22:24:47.604940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:47.604450 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" event={"ID":"00258507-e531-4b1f-a3ad-fa55c532c503","Type":"ContainerDied","Data":"0811a8941f0fe9a56123d51ba59bdc97071dd05e5eced04b2d4052f110bafc3d"} Nov 02 22:24:48.025119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:48.025032 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq"] Nov 02 22:24:48.032114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:48.032079 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w"] Nov 02 22:24:48.037421 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:48.037397 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb47675b-jv7pq"] Nov 02 22:24:48.041003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:48.040980 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe2d002-k7f7w"] Nov 02 22:24:48.270772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:48.266880 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e621f94-bf4b-41de-956a-700fa5f5af20" path="/var/lib/kubelet/pods/2e621f94-bf4b-41de-956a-700fa5f5af20/volumes" Nov 02 22:24:48.270772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:48.267367 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb41b1fe-7c1f-45da-bd49-a7ca79ea921e" path="/var/lib/kubelet/pods/bb41b1fe-7c1f-45da-bd49-a7ca79ea921e/volumes" Nov 02 22:24:49.030982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.030918 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:49.100984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.100948 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-openshift-ca\") pod \"00258507-e531-4b1f-a3ad-fa55c532c503\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " Nov 02 22:24:49.101185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.101166 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-xks-ca\") pod \"00258507-e531-4b1f-a3ad-fa55c532c503\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " Nov 02 22:24:49.101266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.101254 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00258507-e531-4b1f-a3ad-fa55c532c503-kube-api-access-skdgw\" (UniqueName: \"kubernetes.io/projected/00258507-e531-4b1f-a3ad-fa55c532c503-kube-api-access-skdgw\") pod \"00258507-e531-4b1f-a3ad-fa55c532c503\" (UID: \"00258507-e531-4b1f-a3ad-fa55c532c503\") " Nov 02 22:24:49.115284 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.108863 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-openshift-ca" pod "00258507-e531-4b1f-a3ad-fa55c532c503" (UID: "00258507-e531-4b1f-a3ad-fa55c532c503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:49.116779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.116750 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00258507-e531-4b1f-a3ad-fa55c532c503-kube-api-access-skdgw" pod "00258507-e531-4b1f-a3ad-fa55c532c503" (UID: "00258507-e531-4b1f-a3ad-fa55c532c503"). InnerVolumeSpecName "kube-api-access-skdgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:49.125137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.125099 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-xks-ca" pod "00258507-e531-4b1f-a3ad-fa55c532c503" (UID: "00258507-e531-4b1f-a3ad-fa55c532c503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:24:49.204291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.204256 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:49.204291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.204294 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdgw\" (UniqueName: \"kubernetes.io/projected/00258507-e531-4b1f-a3ad-fa55c532c503-kube-api-access-skdgw\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:49.204480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.204310 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00258507-e531-4b1f-a3ad-fa55c532c503-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:49.707943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.707910 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" Nov 02 22:24:49.708142 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.707943 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll" event={"ID":"00258507-e531-4b1f-a3ad-fa55c532c503","Type":"ContainerDied","Data":"98c4a4d8baeb973028764e08feb1681e172760a17cb61bbcc14945be8fb59198"} Nov 02 22:24:49.708142 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:49.707992 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98c4a4d8baeb973028764e08feb1681e172760a17cb61bbcc14945be8fb59198" Nov 02 22:24:51.795186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.792778 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2"] Nov 02 22:24:51.795186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.793946 4309 cpu_manager.go:414] "containerMap: removing container" podUID="00258507-e531-4b1f-a3ad-fa55c532c503" containerName="revoke-keys" Nov 02 22:24:51.795186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.793966 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00258507-e531-4b1f-a3ad-fa55c532c503" containerName="revoke-keys" Nov 02 22:24:51.795186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.794199 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="00258507-e531-4b1f-a3ad-fa55c532c503" containerName="revoke-keys" Nov 02 22:24:51.799695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.799026 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:51.807699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.802422 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:24:51.807699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.805929 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:24:51.807699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.806028 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:24:51.807699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.806033 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:24:51.809032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.808972 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2"] Nov 02 22:24:51.908635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.908599 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8npr\" (UniqueName: \"kubernetes.io/projected/1d1c3a1b-62e3-462f-9349-16d2665af0c2-kube-api-access-q8npr\") pod \"maas-api-revoke-sub-a329dc5c-p42w2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:51.908856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.908708 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a329dc5c-p42w2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:51.908856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:51.908791 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a329dc5c-p42w2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:52.010705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.010634 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8npr\" (UniqueName: \"kubernetes.io/projected/1d1c3a1b-62e3-462f-9349-16d2665af0c2-kube-api-access-q8npr\") pod \"maas-api-revoke-sub-a329dc5c-p42w2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:52.010938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.010764 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a329dc5c-p42w2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:52.010938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.010796 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a329dc5c-p42w2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:52.011576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.011548 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a329dc5c-p42w2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:52.013322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.013297 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a329dc5c-p42w2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:52.018624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.018597 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8npr\" (UniqueName: \"kubernetes.io/projected/1d1c3a1b-62e3-462f-9349-16d2665af0c2-kube-api-access-q8npr\") pod \"maas-api-revoke-sub-a329dc5c-p42w2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:52.129283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.129253 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:52.421935 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:24:52.421901 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1c3a1b_62e3_462f_9349_16d2665af0c2.slice/crio-a949cbe95cacc2b3c5d5834eacee5dc4ae4600139dcb28951adca9a50eeef1d1 WatchSource:0}: Error finding container a949cbe95cacc2b3c5d5834eacee5dc4ae4600139dcb28951adca9a50eeef1d1: Status 404 returned error can't find the container with id a949cbe95cacc2b3c5d5834eacee5dc4ae4600139dcb28951adca9a50eeef1d1 Nov 02 22:24:52.422297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.422274 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2"] Nov 02 22:24:52.734365 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.734312 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" event={"ID":"1d1c3a1b-62e3-462f-9349-16d2665af0c2","Type":"ContainerStarted","Data":"ee1a12a2065550297c6a249c6b8163b80062b2816a87e0014c7a5ed92db04acf"} Nov 02 22:24:52.734545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.734390 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" event={"ID":"1d1c3a1b-62e3-462f-9349-16d2665af0c2","Type":"ContainerStarted","Data":"a949cbe95cacc2b3c5d5834eacee5dc4ae4600139dcb28951adca9a50eeef1d1"} Nov 02 22:24:52.755251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:52.755205 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" podStartSLOduration=1.755190946 podStartE2EDuration="1.755190946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:24:52.749234104 +0000 UTC m=+1620.687347322" watchObservedRunningTime="2026-11-02 22:24:52.755190946 +0000 UTC m=+1620.693304173" Nov 02 22:24:53.808753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:53.799019 4309 generic.go:356] "Generic (PLEG): container finished" podID="1d1c3a1b-62e3-462f-9349-16d2665af0c2" containerID="ee1a12a2065550297c6a249c6b8163b80062b2816a87e0014c7a5ed92db04acf" exitCode=0 Nov 02 22:24:53.808753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:53.799128 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" event={"ID":"1d1c3a1b-62e3-462f-9349-16d2665af0c2","Type":"ContainerDied","Data":"ee1a12a2065550297c6a249c6b8163b80062b2816a87e0014c7a5ed92db04acf"} Nov 02 22:24:53.932758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:53.932717 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f"] Nov 02 22:24:53.943413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:53.941308 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-419f68-tdp5f"] Nov 02 22:24:54.225748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:54.224692 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2eab28f-7f09-4e24-a291-6141cb5a5560" path="/var/lib/kubelet/pods/d2eab28f-7f09-4e24-a291-6141cb5a5560/volumes" Nov 02 22:24:55.039172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.039145 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:55.194823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.194739 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-openshift-ca\") pod \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " Nov 02 22:24:55.195012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.194887 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-xks-ca\") pod \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " Nov 02 22:24:55.195012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.194959 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1c3a1b-62e3-462f-9349-16d2665af0c2-kube-api-access-q8npr\" (UniqueName: \"kubernetes.io/projected/1d1c3a1b-62e3-462f-9349-16d2665af0c2-kube-api-access-q8npr\") pod \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\" (UID: \"1d1c3a1b-62e3-462f-9349-16d2665af0c2\") " Nov 02 22:24:55.195189 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.195164 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-openshift-ca" pod "1d1c3a1b-62e3-462f-9349-16d2665af0c2" (UID: "1d1c3a1b-62e3-462f-9349-16d2665af0c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:24:55.195450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.195434 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:55.196978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.196949 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-xks-ca" pod "1d1c3a1b-62e3-462f-9349-16d2665af0c2" (UID: "1d1c3a1b-62e3-462f-9349-16d2665af0c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:24:55.197285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.197265 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1c3a1b-62e3-462f-9349-16d2665af0c2-kube-api-access-q8npr" pod "1d1c3a1b-62e3-462f-9349-16d2665af0c2" (UID: "1d1c3a1b-62e3-462f-9349-16d2665af0c2"). InnerVolumeSpecName "kube-api-access-q8npr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:24:55.297760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.297728 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d1c3a1b-62e3-462f-9349-16d2665af0c2-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:55.297760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.297756 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8npr\" (UniqueName: \"kubernetes.io/projected/1d1c3a1b-62e3-462f-9349-16d2665af0c2-kube-api-access-q8npr\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:24:55.820406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.820374 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" Nov 02 22:24:55.820558 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.820410 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2" event={"ID":"1d1c3a1b-62e3-462f-9349-16d2665af0c2","Type":"ContainerDied","Data":"a949cbe95cacc2b3c5d5834eacee5dc4ae4600139dcb28951adca9a50eeef1d1"} Nov 02 22:24:55.820558 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:55.820463 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a949cbe95cacc2b3c5d5834eacee5dc4ae4600139dcb28951adca9a50eeef1d1" Nov 02 22:24:58.026057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:58.026008 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd"] Nov 02 22:24:58.030358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:58.030334 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9"] Nov 02 22:24:58.033905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:58.033884 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2051633e-5m2kd"] Nov 02 22:24:58.037668 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:58.037645 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f654977b-q2bd9"] Nov 02 22:24:58.193624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:58.193592 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c172f3-cb47-4972-b252-908b26ef5616" path="/var/lib/kubelet/pods/06c172f3-cb47-4972-b252-908b26ef5616/volumes" Nov 02 22:24:58.194295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:58.194276 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d81db8-ebfe-4ec8-80f6-b93face79b11" path="/var/lib/kubelet/pods/b5d81db8-ebfe-4ec8-80f6-b93face79b11/volumes" Nov 02 22:24:59.024732 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.024694 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf"] Nov 02 22:24:59.025965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.025936 4309 cpu_manager.go:414] "containerMap: removing container" podUID="1d1c3a1b-62e3-462f-9349-16d2665af0c2" containerName="revoke-keys" Nov 02 22:24:59.025965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.025961 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1c3a1b-62e3-462f-9349-16d2665af0c2" containerName="revoke-keys" Nov 02 22:24:59.026124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.026120 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1c3a1b-62e3-462f-9349-16d2665af0c2" containerName="revoke-keys" Nov 02 22:24:59.031513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.031491 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.038809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.038787 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:24:59.040127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.040104 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:24:59.040291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.040109 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:24:59.040291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.040172 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:24:59.042902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.042879 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf"] Nov 02 22:24:59.094757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.094726 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be757162-qn8gf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.094931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.094765 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be757162-qn8gf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.095020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.094973 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcz8v\" (UniqueName: \"kubernetes.io/projected/f23bf9d1-a705-467c-b7b2-d955c986b4bf-kube-api-access-kcz8v\") pod \"maas-api-revoke-sub-be757162-qn8gf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.196285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.196250 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be757162-qn8gf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.196470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.196289 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be757162-qn8gf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.196516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.196473 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcz8v\" (UniqueName: \"kubernetes.io/projected/f23bf9d1-a705-467c-b7b2-d955c986b4bf-kube-api-access-kcz8v\") pod \"maas-api-revoke-sub-be757162-qn8gf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.197072 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.197046 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be757162-qn8gf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.198625 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.198604 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be757162-qn8gf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.204505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.204476 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcz8v\" (UniqueName: \"kubernetes.io/projected/f23bf9d1-a705-467c-b7b2-d955c986b4bf-kube-api-access-kcz8v\") pod \"maas-api-revoke-sub-be757162-qn8gf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.353227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.353176 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:24:59.699063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.699026 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf"] Nov 02 22:24:59.700118 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:24:59.700088 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23bf9d1_a705_467c_b7b2_d955c986b4bf.slice/crio-68f8a7d4534e5cb8f05116a5d78363b43752950af1e0fc91b3ed3fb9ab39672f WatchSource:0}: Error finding container 68f8a7d4534e5cb8f05116a5d78363b43752950af1e0fc91b3ed3fb9ab39672f: Status 404 returned error can't find the container with id 68f8a7d4534e5cb8f05116a5d78363b43752950af1e0fc91b3ed3fb9ab39672f Nov 02 22:24:59.837359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.837324 4309 scope.go:122] "RemoveContainer" containerID="9d2a2dc6b1379a1d274ce79546a1a03ca2158c848300d1f4751c40a5fed8a8af" Nov 02 22:24:59.848331 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.848312 4309 scope.go:122] "RemoveContainer" containerID="b9f8dd0e5aae76f085d10e45148dad86ccbe6268f4bc08479b614232c4bac390" Nov 02 22:24:59.861424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.861402 4309 scope.go:122] "RemoveContainer" containerID="381f220b0bccb20bfc0242f1cfc6dec493469fcb808067785cf3681eb95a450d" Nov 02 22:24:59.906931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:24:59.906897 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" event={"ID":"f23bf9d1-a705-467c-b7b2-d955c986b4bf","Type":"ContainerStarted","Data":"68f8a7d4534e5cb8f05116a5d78363b43752950af1e0fc91b3ed3fb9ab39672f"} Nov 02 22:25:00.921599 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:00.921500 4309 generic.go:356] "Generic (PLEG): container finished" podID="f23bf9d1-a705-467c-b7b2-d955c986b4bf" containerID="ff4e27bd8ac0d5ed2c52a6dd497a8e49a7b8330a7f30f69799899516a3eb937c" exitCode=0 Nov 02 22:25:00.921985 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:00.921733 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" event={"ID":"f23bf9d1-a705-467c-b7b2-d955c986b4bf","Type":"ContainerDied","Data":"ff4e27bd8ac0d5ed2c52a6dd497a8e49a7b8330a7f30f69799899516a3eb937c"} Nov 02 22:25:01.057836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.057806 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642"] Nov 02 22:25:01.058321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.058220 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" containerID="cri-o://db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916" gracePeriod=30 Nov 02 22:25:01.205727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.202762 4309 patch_prober.go:29] interesting pod/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:25:01.205727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.202850 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:25:01.531359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.531262 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb"] Nov 02 22:25:01.536758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.536731 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.543224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.543193 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb"] Nov 02 22:25:01.606274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.606234 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.606516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.606483 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6l5s\" (UniqueName: \"kubernetes.io/projected/d21d18fe-86f1-4332-93b2-f244d584602a-kube-api-access-t6l5s\") pod \"maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.606658 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.606601 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.717797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.712445 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6l5s\" (UniqueName: \"kubernetes.io/projected/d21d18fe-86f1-4332-93b2-f244d584602a-kube-api-access-t6l5s\") pod \"maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.717797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.712594 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.732697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.731492 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.769369 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.760656 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.769369 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.766916 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.788751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.778023 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6l5s\" (UniqueName: \"kubernetes.io/projected/d21d18fe-86f1-4332-93b2-f244d584602a-kube-api-access-t6l5s\") pod \"maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:01.858315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:01.858279 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:02.139931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.139902 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:25:02.149763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.149730 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb"] Nov 02 22:25:02.150591 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:02.150569 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21d18fe_86f1_4332_93b2_f244d584602a.slice/crio-20092b0da8651c49a32901694ae5a7f29a7b47915df68020fb1d308506bd4d7d WatchSource:0}: Error finding container 20092b0da8651c49a32901694ae5a7f29a7b47915df68020fb1d308506bd4d7d: Status 404 returned error can't find the container with id 20092b0da8651c49a32901694ae5a7f29a7b47915df68020fb1d308506bd4d7d Nov 02 22:25:02.240509 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.240476 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-xks-ca\") pod \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " Nov 02 22:25:02.240709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.240560 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-openshift-ca\") pod \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " Nov 02 22:25:02.240761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.240732 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23bf9d1-a705-467c-b7b2-d955c986b4bf-kube-api-access-kcz8v\" (UniqueName: \"kubernetes.io/projected/f23bf9d1-a705-467c-b7b2-d955c986b4bf-kube-api-access-kcz8v\") pod \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\" (UID: \"f23bf9d1-a705-467c-b7b2-d955c986b4bf\") " Nov 02 22:25:02.241002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.240973 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-openshift-ca" pod "f23bf9d1-a705-467c-b7b2-d955c986b4bf" (UID: "f23bf9d1-a705-467c-b7b2-d955c986b4bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:02.241278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.241261 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:02.282841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.282806 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-xks-ca" pod "f23bf9d1-a705-467c-b7b2-d955c986b4bf" (UID: "f23bf9d1-a705-467c-b7b2-d955c986b4bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:02.283023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.282943 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23bf9d1-a705-467c-b7b2-d955c986b4bf-kube-api-access-kcz8v" pod "f23bf9d1-a705-467c-b7b2-d955c986b4bf" (UID: "f23bf9d1-a705-467c-b7b2-d955c986b4bf"). InnerVolumeSpecName "kube-api-access-kcz8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:02.343188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.343144 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcz8v\" (UniqueName: \"kubernetes.io/projected/f23bf9d1-a705-467c-b7b2-d955c986b4bf-kube-api-access-kcz8v\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:02.343188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.343184 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23bf9d1-a705-467c-b7b2-d955c986b4bf-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:02.945419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.945387 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" Nov 02 22:25:02.945419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.945415 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf" event={"ID":"f23bf9d1-a705-467c-b7b2-d955c986b4bf","Type":"ContainerDied","Data":"68f8a7d4534e5cb8f05116a5d78363b43752950af1e0fc91b3ed3fb9ab39672f"} Nov 02 22:25:02.945667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.945462 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68f8a7d4534e5cb8f05116a5d78363b43752950af1e0fc91b3ed3fb9ab39672f" Nov 02 22:25:02.947391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.947364 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" event={"ID":"d21d18fe-86f1-4332-93b2-f244d584602a","Type":"ContainerStarted","Data":"d82ee1eb13997e90abcd8b6188300e947c100e9f42423c9253ad0459366cf99f"} Nov 02 22:25:02.947527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.947410 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" event={"ID":"d21d18fe-86f1-4332-93b2-f244d584602a","Type":"ContainerStarted","Data":"20092b0da8651c49a32901694ae5a7f29a7b47915df68020fb1d308506bd4d7d"} Nov 02 22:25:02.966251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:02.966200 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" podStartSLOduration=1.966185221 podStartE2EDuration="1.966185221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:02.963834124 +0000 UTC m=+1630.901947346" watchObservedRunningTime="2026-11-02 22:25:02.966185221 +0000 UTC m=+1630.904298438" Nov 02 22:25:03.957373 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:03.957342 4309 generic.go:356] "Generic (PLEG): container finished" podID="d21d18fe-86f1-4332-93b2-f244d584602a" containerID="d82ee1eb13997e90abcd8b6188300e947c100e9f42423c9253ad0459366cf99f" exitCode=0 Nov 02 22:25:03.957812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:03.957430 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" event={"ID":"d21d18fe-86f1-4332-93b2-f244d584602a","Type":"ContainerDied","Data":"d82ee1eb13997e90abcd8b6188300e947c100e9f42423c9253ad0459366cf99f"} Nov 02 22:25:04.443760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.443663 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r"] Nov 02 22:25:04.445547 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.445506 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f23bf9d1-a705-467c-b7b2-d955c986b4bf" containerName="revoke-keys" Nov 02 22:25:04.445720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.445557 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23bf9d1-a705-467c-b7b2-d955c986b4bf" containerName="revoke-keys" Nov 02 22:25:04.446100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.446077 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23bf9d1-a705-467c-b7b2-d955c986b4bf" containerName="revoke-keys" Nov 02 22:25:04.454012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.453950 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.461839 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.461807 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r"] Nov 02 22:25:04.482434 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.482403 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-982b8836-tkc2r\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.482586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.482480 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-982b8836-tkc2r\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.482716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.482662 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7k7\" (UniqueName: \"kubernetes.io/projected/3a11a862-1ca2-43d5-9847-46e7113b9908-kube-api-access-md7k7\") pod \"maas-api-revoke-sub-982b8836-tkc2r\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.585697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.584710 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-982b8836-tkc2r\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.585697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.584798 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-982b8836-tkc2r\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.585697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.584910 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7k7\" (UniqueName: \"kubernetes.io/projected/3a11a862-1ca2-43d5-9847-46e7113b9908-kube-api-access-md7k7\") pod \"maas-api-revoke-sub-982b8836-tkc2r\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.617697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.615531 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-982b8836-tkc2r\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.650556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.649274 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-xks-ca\") pod \"maas-api-revoke-sub-982b8836-tkc2r\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.650556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.650033 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7k7\" (UniqueName: \"kubernetes.io/projected/3a11a862-1ca2-43d5-9847-46e7113b9908-kube-api-access-md7k7\") pod \"maas-api-revoke-sub-982b8836-tkc2r\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:04.781479 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:04.781406 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:05.079010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.078978 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r"] Nov 02 22:25:05.097693 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:05.092198 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a11a862_1ca2_43d5_9847_46e7113b9908.slice/crio-310ac0cd9c61b4c7a9261a4f00558a685119f456f759c62e020338605762b8f6 WatchSource:0}: Error finding container 310ac0cd9c61b4c7a9261a4f00558a685119f456f759c62e020338605762b8f6: Status 404 returned error can't find the container with id 310ac0cd9c61b4c7a9261a4f00558a685119f456f759c62e020338605762b8f6 Nov 02 22:25:05.131099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.131075 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:05.309377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.309341 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-openshift-ca\") pod \"d21d18fe-86f1-4332-93b2-f244d584602a\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " Nov 02 22:25:05.309582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.309467 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-xks-ca\") pod \"d21d18fe-86f1-4332-93b2-f244d584602a\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " Nov 02 22:25:05.309582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.309520 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21d18fe-86f1-4332-93b2-f244d584602a-kube-api-access-t6l5s\" (UniqueName: \"kubernetes.io/projected/d21d18fe-86f1-4332-93b2-f244d584602a-kube-api-access-t6l5s\") pod \"d21d18fe-86f1-4332-93b2-f244d584602a\" (UID: \"d21d18fe-86f1-4332-93b2-f244d584602a\") " Nov 02 22:25:05.310073 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.309912 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-openshift-ca" pod "d21d18fe-86f1-4332-93b2-f244d584602a" (UID: "d21d18fe-86f1-4332-93b2-f244d584602a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:05.311111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.311079 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:05.321456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.321094 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21d18fe-86f1-4332-93b2-f244d584602a-kube-api-access-t6l5s" pod "d21d18fe-86f1-4332-93b2-f244d584602a" (UID: "d21d18fe-86f1-4332-93b2-f244d584602a"). InnerVolumeSpecName "kube-api-access-t6l5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:05.321667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.321626 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-xks-ca" pod "d21d18fe-86f1-4332-93b2-f244d584602a" (UID: "d21d18fe-86f1-4332-93b2-f244d584602a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:05.412710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.412659 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21d18fe-86f1-4332-93b2-f244d584602a-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:05.412913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.412715 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6l5s\" (UniqueName: \"kubernetes.io/projected/d21d18fe-86f1-4332-93b2-f244d584602a-kube-api-access-t6l5s\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:05.573698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.570874 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp"] Nov 02 22:25:05.573698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.572245 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d21d18fe-86f1-4332-93b2-f244d584602a" containerName="revoke-keys" Nov 02 22:25:05.573698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.572268 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21d18fe-86f1-4332-93b2-f244d584602a" containerName="revoke-keys" Nov 02 22:25:05.573698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.572465 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21d18fe-86f1-4332-93b2-f244d584602a" containerName="revoke-keys" Nov 02 22:25:05.601415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.582754 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.701584 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.701548 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp"] Nov 02 22:25:05.736354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.736301 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.736564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.736486 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6thj\" (UniqueName: \"kubernetes.io/projected/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-kube-api-access-j6thj\") pod \"maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.736856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.736832 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.838502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.838469 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.838690 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.838566 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.838760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.838691 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6thj\" (UniqueName: \"kubernetes.io/projected/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-kube-api-access-j6thj\") pod \"maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.839296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.839271 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.844405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.844378 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.846477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.846452 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6thj\" (UniqueName: \"kubernetes.io/projected/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-kube-api-access-j6thj\") pod \"maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.976507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.976416 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" Nov 02 22:25:05.976507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.976447 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb" event={"ID":"d21d18fe-86f1-4332-93b2-f244d584602a","Type":"ContainerDied","Data":"20092b0da8651c49a32901694ae5a7f29a7b47915df68020fb1d308506bd4d7d"} Nov 02 22:25:05.976507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.976498 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20092b0da8651c49a32901694ae5a7f29a7b47915df68020fb1d308506bd4d7d" Nov 02 22:25:05.978232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.978203 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" event={"ID":"3a11a862-1ca2-43d5-9847-46e7113b9908","Type":"ContainerStarted","Data":"d2e704953dc6afd0a4448cee62d60d76e44d5cf72db10970ceddb96a7220e895"} Nov 02 22:25:05.978358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.978253 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" event={"ID":"3a11a862-1ca2-43d5-9847-46e7113b9908","Type":"ContainerStarted","Data":"310ac0cd9c61b4c7a9261a4f00558a685119f456f759c62e020338605762b8f6"} Nov 02 22:25:05.980965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.980947 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:05.997207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:05.997150 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" podStartSLOduration=1.9971313689999999 podStartE2EDuration="1.997131369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:05.992523166 +0000 UTC m=+1633.930636388" watchObservedRunningTime="2026-11-02 22:25:05.997131369 +0000 UTC m=+1633.935244590" Nov 02 22:25:06.283482 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:06.283444 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb3d6ea_9ee0_404d_a41a_ed026d70a903.slice/crio-496f1b0702475571946d0ab4698abd80ad4b7263260d53d48842e13b60ccb8bf WatchSource:0}: Error finding container 496f1b0702475571946d0ab4698abd80ad4b7263260d53d48842e13b60ccb8bf: Status 404 returned error can't find the container with id 496f1b0702475571946d0ab4698abd80ad4b7263260d53d48842e13b60ccb8bf Nov 02 22:25:06.285799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.285292 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp"] Nov 02 22:25:06.344550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.344528 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:25:06.494046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.485301 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-data\") pod \"8e3c8d49-94e7-402f-945c-2888cf3a8930\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " Nov 02 22:25:06.494046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.485493 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-credential-socket\") pod \"8e3c8d49-94e7-402f-945c-2888cf3a8930\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " Nov 02 22:25:06.494046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.485566 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-envoy\") pod \"8e3c8d49-94e7-402f-945c-2888cf3a8930\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " Nov 02 22:25:06.494046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.485618 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3c8d49-94e7-402f-945c-2888cf3a8930-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c8d49-94e7-402f-945c-2888cf3a8930-istiod-ca-cert\") pod \"8e3c8d49-94e7-402f-945c-2888cf3a8930\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " Nov 02 22:25:06.494046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.485768 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-podinfo\") pod \"8e3c8d49-94e7-402f-945c-2888cf3a8930\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " Nov 02 22:25:06.494046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.485935 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-socket\") pod \"8e3c8d49-94e7-402f-945c-2888cf3a8930\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " Nov 02 22:25:06.494046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.486010 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-token\") pod \"8e3c8d49-94e7-402f-945c-2888cf3a8930\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " Nov 02 22:25:06.494046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.486240 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-kube-api-access-8svhs\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-kube-api-access-8svhs\") pod \"8e3c8d49-94e7-402f-945c-2888cf3a8930\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " Nov 02 22:25:06.494046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.486331 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-certs\") pod \"8e3c8d49-94e7-402f-945c-2888cf3a8930\" (UID: \"8e3c8d49-94e7-402f-945c-2888cf3a8930\") " Nov 02 22:25:06.495351 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.494988 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-certs" pod "8e3c8d49-94e7-402f-945c-2888cf3a8930" (UID: "8e3c8d49-94e7-402f-945c-2888cf3a8930"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:06.495802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.495521 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-socket" pod "8e3c8d49-94e7-402f-945c-2888cf3a8930" (UID: "8e3c8d49-94e7-402f-945c-2888cf3a8930"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:06.496281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.496136 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-credential-socket" pod "8e3c8d49-94e7-402f-945c-2888cf3a8930" (UID: "8e3c8d49-94e7-402f-945c-2888cf3a8930"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:06.496651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.496409 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-data" pod "8e3c8d49-94e7-402f-945c-2888cf3a8930" (UID: "8e3c8d49-94e7-402f-945c-2888cf3a8930"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:06.501106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.498863 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3c8d49-94e7-402f-945c-2888cf3a8930-istiod-ca-cert" pod "8e3c8d49-94e7-402f-945c-2888cf3a8930" (UID: "8e3c8d49-94e7-402f-945c-2888cf3a8930"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:06.512357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.512305 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-envoy" pod "8e3c8d49-94e7-402f-945c-2888cf3a8930" (UID: "8e3c8d49-94e7-402f-945c-2888cf3a8930"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:06.512647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.512611 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-podinfo" pod "8e3c8d49-94e7-402f-945c-2888cf3a8930" (UID: "8e3c8d49-94e7-402f-945c-2888cf3a8930"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:06.521694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.518165 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-kube-api-access-8svhs" pod "8e3c8d49-94e7-402f-945c-2888cf3a8930" (UID: "8e3c8d49-94e7-402f-945c-2888cf3a8930"). InnerVolumeSpecName "kube-api-access-8svhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:06.521694 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.518206 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-token" pod "8e3c8d49-94e7-402f-945c-2888cf3a8930" (UID: "8e3c8d49-94e7-402f-945c-2888cf3a8930"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:06.588337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.588299 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:06.588337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.588324 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:06.588337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.588339 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3c8d49-94e7-402f-945c-2888cf3a8930-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:06.588637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.588354 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:06.588637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.588364 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:06.588637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.588374 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:06.588637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.588387 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8svhs\" (UniqueName: \"kubernetes.io/projected/8e3c8d49-94e7-402f-945c-2888cf3a8930-kube-api-access-8svhs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:06.588637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.588397 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:06.588637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.588405 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3c8d49-94e7-402f-945c-2888cf3a8930-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:06.990028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.989984 4309 generic.go:356] "Generic (PLEG): container finished" podID="3a11a862-1ca2-43d5-9847-46e7113b9908" containerID="d2e704953dc6afd0a4448cee62d60d76e44d5cf72db10970ceddb96a7220e895" exitCode=0 Nov 02 22:25:06.990225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.990100 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" event={"ID":"3a11a862-1ca2-43d5-9847-46e7113b9908","Type":"ContainerDied","Data":"d2e704953dc6afd0a4448cee62d60d76e44d5cf72db10970ceddb96a7220e895"} Nov 02 22:25:06.992153 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.992126 4309 generic.go:356] "Generic (PLEG): container finished" podID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerID="db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916" exitCode=0 Nov 02 22:25:06.992305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.992183 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" Nov 02 22:25:06.992305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.992197 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" event={"ID":"8e3c8d49-94e7-402f-945c-2888cf3a8930","Type":"ContainerDied","Data":"db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916"} Nov 02 22:25:06.992305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.992239 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642" event={"ID":"8e3c8d49-94e7-402f-945c-2888cf3a8930","Type":"ContainerDied","Data":"aaf68813b2ebab7f7e32b8ff66bf4133c02eefe9493e192d5f7eff22bc80561d"} Nov 02 22:25:06.992305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.992259 4309 scope.go:122] "RemoveContainer" containerID="db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916" Nov 02 22:25:06.995356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.995296 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" event={"ID":"ebb3d6ea-9ee0-404d-a41a-ed026d70a903","Type":"ContainerStarted","Data":"e4600c1903344d21a2c4fa68c7b22f4784aa6af00c6f58bce6d5edc830e97313"} Nov 02 22:25:06.995356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:06.995349 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" event={"ID":"ebb3d6ea-9ee0-404d-a41a-ed026d70a903","Type":"ContainerStarted","Data":"496f1b0702475571946d0ab4698abd80ad4b7263260d53d48842e13b60ccb8bf"} Nov 02 22:25:07.007304 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:07.007271 4309 scope.go:122] "RemoveContainer" containerID="1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962" Nov 02 22:25:07.022460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:07.022433 4309 scope.go:122] "RemoveContainer" containerID="db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916" Nov 02 22:25:07.024360 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:07.024328 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916\": container with ID starting with db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916 not found: ID does not exist" containerID="db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916" Nov 02 22:25:07.024505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:07.024370 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916"} err="failed to get container status \"db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916\": rpc error: code = NotFound desc = could not find container \"db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916\": container with ID starting with db2b297ea43da1a3ea71b00f3027ac36b984fd748073c3bcf6c16ee8da847916 not found: ID does not exist" Nov 02 22:25:07.024505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:07.024395 4309 scope.go:122] "RemoveContainer" containerID="1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962" Nov 02 22:25:07.024821 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:07.024793 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962\": container with ID starting with 1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962 not found: ID does not exist" containerID="1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962" Nov 02 22:25:07.024954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:07.024827 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962"} err="failed to get container status \"1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962\": rpc error: code = NotFound desc = could not find container \"1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962\": container with ID starting with 1b5591157edbd565ee9beec44b830ef8d0a0726b244395ae98a802869c09a962 not found: ID does not exist" Nov 02 22:25:07.025809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:07.025786 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642"] Nov 02 22:25:07.030204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:07.030180 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-419f68-openshift-default-7fbf675875-bv642"] Nov 02 22:25:07.038645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:07.038604 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" podStartSLOduration=2.038589686 podStartE2EDuration="2.038589686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:07.03714776 +0000 UTC m=+1634.975260979" watchObservedRunningTime="2026-11-02 22:25:07.038589686 +0000 UTC m=+1634.976702906" Nov 02 22:25:08.019844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.019803 4309 generic.go:356] "Generic (PLEG): container finished" podID="ebb3d6ea-9ee0-404d-a41a-ed026d70a903" containerID="e4600c1903344d21a2c4fa68c7b22f4784aa6af00c6f58bce6d5edc830e97313" exitCode=0 Nov 02 22:25:08.020481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.020453 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" event={"ID":"ebb3d6ea-9ee0-404d-a41a-ed026d70a903","Type":"ContainerDied","Data":"e4600c1903344d21a2c4fa68c7b22f4784aa6af00c6f58bce6d5edc830e97313"} Nov 02 22:25:08.192928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.192899 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" path="/var/lib/kubelet/pods/8e3c8d49-94e7-402f-945c-2888cf3a8930/volumes" Nov 02 22:25:08.227968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.227800 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:08.328507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.328468 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-openshift-ca\") pod \"3a11a862-1ca2-43d5-9847-46e7113b9908\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " Nov 02 22:25:08.328662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.328606 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a11a862-1ca2-43d5-9847-46e7113b9908-kube-api-access-md7k7\" (UniqueName: \"kubernetes.io/projected/3a11a862-1ca2-43d5-9847-46e7113b9908-kube-api-access-md7k7\") pod \"3a11a862-1ca2-43d5-9847-46e7113b9908\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " Nov 02 22:25:08.328754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.328709 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-xks-ca\") pod \"3a11a862-1ca2-43d5-9847-46e7113b9908\" (UID: \"3a11a862-1ca2-43d5-9847-46e7113b9908\") " Nov 02 22:25:08.329016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.328875 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-openshift-ca" pod "3a11a862-1ca2-43d5-9847-46e7113b9908" (UID: "3a11a862-1ca2-43d5-9847-46e7113b9908"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:08.329653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.329618 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:08.330916 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.330893 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a11a862-1ca2-43d5-9847-46e7113b9908-kube-api-access-md7k7" pod "3a11a862-1ca2-43d5-9847-46e7113b9908" (UID: "3a11a862-1ca2-43d5-9847-46e7113b9908"). InnerVolumeSpecName "kube-api-access-md7k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:08.331081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.331056 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-xks-ca" pod "3a11a862-1ca2-43d5-9847-46e7113b9908" (UID: "3a11a862-1ca2-43d5-9847-46e7113b9908"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:08.431199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.431161 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md7k7\" (UniqueName: \"kubernetes.io/projected/3a11a862-1ca2-43d5-9847-46e7113b9908-kube-api-access-md7k7\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:08.431199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:08.431197 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a11a862-1ca2-43d5-9847-46e7113b9908-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:09.031269 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.031233 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" event={"ID":"3a11a862-1ca2-43d5-9847-46e7113b9908","Type":"ContainerDied","Data":"310ac0cd9c61b4c7a9261a4f00558a685119f456f759c62e020338605762b8f6"} Nov 02 22:25:09.031838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.031284 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="310ac0cd9c61b4c7a9261a4f00558a685119f456f759c62e020338605762b8f6" Nov 02 22:25:09.031838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.031356 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r" Nov 02 22:25:09.265167 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.265138 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:09.353487 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.353457 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-kube-api-access-j6thj\" (UniqueName: \"kubernetes.io/projected/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-kube-api-access-j6thj\") pod \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " Nov 02 22:25:09.353697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.353516 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-xks-ca\") pod \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " Nov 02 22:25:09.353697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.353561 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-openshift-ca\") pod \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\" (UID: \"ebb3d6ea-9ee0-404d-a41a-ed026d70a903\") " Nov 02 22:25:09.354016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.353990 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-openshift-ca" pod "ebb3d6ea-9ee0-404d-a41a-ed026d70a903" (UID: "ebb3d6ea-9ee0-404d-a41a-ed026d70a903"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:09.354288 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.354269 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:09.355635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.355607 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-kube-api-access-j6thj" pod "ebb3d6ea-9ee0-404d-a41a-ed026d70a903" (UID: "ebb3d6ea-9ee0-404d-a41a-ed026d70a903"). InnerVolumeSpecName "kube-api-access-j6thj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:09.355635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.355609 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-xks-ca" pod "ebb3d6ea-9ee0-404d-a41a-ed026d70a903" (UID: "ebb3d6ea-9ee0-404d-a41a-ed026d70a903"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:09.469718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.459461 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6thj\" (UniqueName: \"kubernetes.io/projected/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-kube-api-access-j6thj\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:09.469718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:09.459504 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb3d6ea-9ee0-404d-a41a-ed026d70a903-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:10.126700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:10.126646 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" event={"ID":"ebb3d6ea-9ee0-404d-a41a-ed026d70a903","Type":"ContainerDied","Data":"496f1b0702475571946d0ab4698abd80ad4b7263260d53d48842e13b60ccb8bf"} Nov 02 22:25:10.127233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:10.126727 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="496f1b0702475571946d0ab4698abd80ad4b7263260d53d48842e13b60ccb8bf" Nov 02 22:25:10.127233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:10.126802 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp" Nov 02 22:25:11.602594 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:11.602550 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:25:11.689100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:11.689061 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w"] Nov 02 22:25:11.704626 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:11.704306 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8"] Nov 02 22:25:13.016897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.000177 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr"] Nov 02 22:25:13.020129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.020097 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" Nov 02 22:25:13.020324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.020310 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" Nov 02 22:25:13.020449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.020437 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" Nov 02 22:25:13.020530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.020521 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" Nov 02 22:25:13.020608 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.020599 4309 cpu_manager.go:414] "containerMap: removing container" podUID="3a11a862-1ca2-43d5-9847-46e7113b9908" containerName="revoke-keys" Nov 02 22:25:13.020700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.020689 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a11a862-1ca2-43d5-9847-46e7113b9908" containerName="revoke-keys" Nov 02 22:25:13.020805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.020794 4309 cpu_manager.go:414] "containerMap: removing container" podUID="ebb3d6ea-9ee0-404d-a41a-ed026d70a903" containerName="revoke-keys" Nov 02 22:25:13.020880 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.020869 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb3d6ea-9ee0-404d-a41a-ed026d70a903" containerName="revoke-keys" Nov 02 22:25:13.021203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.021184 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" Nov 02 22:25:13.021328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.021317 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3c8d49-94e7-402f-945c-2888cf3a8930" containerName="istio-proxy" Nov 02 22:25:13.022093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.022073 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a11a862-1ca2-43d5-9847-46e7113b9908" containerName="revoke-keys" Nov 02 22:25:13.022212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.022202 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb3d6ea-9ee0-404d-a41a-ed026d70a903" containerName="revoke-keys" Nov 02 22:25:13.040513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.039975 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr"] Nov 02 22:25:13.040513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.040103 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.052304 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.046615 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:13.052577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.051241 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:25:13.052761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.051279 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:25:13.052888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.051306 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:25:13.193823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.193766 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5mvc\" (UniqueName: \"kubernetes.io/projected/15815a29-6f09-487a-88e6-3014b2361ee1-kube-api-access-n5mvc\") pod \"maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.194005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.193942 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.194057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.194034 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.313753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.311822 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.313753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.312101 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5mvc\" (UniqueName: \"kubernetes.io/projected/15815a29-6f09-487a-88e6-3014b2361ee1-kube-api-access-n5mvc\") pod \"maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.313753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.312297 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.313753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.313316 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.342703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.332215 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.375697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.371827 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5mvc\" (UniqueName: \"kubernetes.io/projected/15815a29-6f09-487a-88e6-3014b2361ee1-kube-api-access-n5mvc\") pod \"maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.416702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.416613 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:13.574211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.574068 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb"] Nov 02 22:25:13.648543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:13.648465 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e49edd-vgnzb"] Nov 02 22:25:14.030723 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:14.030650 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr"] Nov 02 22:25:14.192941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:14.192850 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21d18fe-86f1-4332-93b2-f244d584602a" path="/var/lib/kubelet/pods/d21d18fe-86f1-4332-93b2-f244d584602a/volumes" Nov 02 22:25:14.200636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:14.200597 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" event={"ID":"15815a29-6f09-487a-88e6-3014b2361ee1","Type":"ContainerStarted","Data":"502369d30ed796c3aaea5b573e4e8ba3c409907bb91911e1f1161de05fbef74c"} Nov 02 22:25:15.211922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:15.211885 4309 generic.go:356] "Generic (PLEG): container finished" podID="15815a29-6f09-487a-88e6-3014b2361ee1" containerID="510abe3c6574511a6332c972d96cc24fb8963ffe06d657925e02185c88ab287a" exitCode=0 Nov 02 22:25:15.212430 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:15.211968 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" event={"ID":"15815a29-6f09-487a-88e6-3014b2361ee1","Type":"ContainerDied","Data":"510abe3c6574511a6332c972d96cc24fb8963ffe06d657925e02185c88ab287a"} Nov 02 22:25:15.702330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:15.701260 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd"] Nov 02 22:25:15.915427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:15.915393 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6"] Nov 02 22:25:15.915895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:15.915863 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" containerID="cri-o://d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3" gracePeriod=30 Nov 02 22:25:16.581630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.581601 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:16.690009 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.689955 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" podUID="8f07b490-666a-4652-9c73-b7021c90a5e7" containerName="payload-processing" containerID="cri-o://1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5" gracePeriod=25 Nov 02 22:25:16.705790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.705750 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" podUID="38542064-c103-49d1-9ce6-f0fd1398ac97" containerName="payload-pre-processing" containerID="cri-o://0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa" gracePeriod=25 Nov 02 22:25:16.708166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.708138 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-xks-ca\") pod \"15815a29-6f09-487a-88e6-3014b2361ee1\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " Nov 02 22:25:16.708304 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.708288 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15815a29-6f09-487a-88e6-3014b2361ee1-kube-api-access-n5mvc\" (UniqueName: \"kubernetes.io/projected/15815a29-6f09-487a-88e6-3014b2361ee1-kube-api-access-n5mvc\") pod \"15815a29-6f09-487a-88e6-3014b2361ee1\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " Nov 02 22:25:16.708482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.708462 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-openshift-ca\") pod \"15815a29-6f09-487a-88e6-3014b2361ee1\" (UID: \"15815a29-6f09-487a-88e6-3014b2361ee1\") " Nov 02 22:25:16.709119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.709090 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-openshift-ca" pod "15815a29-6f09-487a-88e6-3014b2361ee1" (UID: "15815a29-6f09-487a-88e6-3014b2361ee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:16.709558 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.709528 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:16.711274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.711249 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-xks-ca" pod "15815a29-6f09-487a-88e6-3014b2361ee1" (UID: "15815a29-6f09-487a-88e6-3014b2361ee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:16.711618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.711593 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15815a29-6f09-487a-88e6-3014b2361ee1-kube-api-access-n5mvc" pod "15815a29-6f09-487a-88e6-3014b2361ee1" (UID: "15815a29-6f09-487a-88e6-3014b2361ee1"). InnerVolumeSpecName "kube-api-access-n5mvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:16.812031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.811925 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15815a29-6f09-487a-88e6-3014b2361ee1-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:16.812031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.811959 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5mvc\" (UniqueName: \"kubernetes.io/projected/15815a29-6f09-487a-88e6-3014b2361ee1-kube-api-access-n5mvc\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:16.983510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:16.983484 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:25:17.116091 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.116007 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f07b490-666a-4652-9c73-b7021c90a5e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f07b490-666a-4652-9c73-b7021c90a5e7-plugins-config-volume\") pod \"8f07b490-666a-4652-9c73-b7021c90a5e7\" (UID: \"8f07b490-666a-4652-9c73-b7021c90a5e7\") " Nov 02 22:25:17.116239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.116193 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f07b490-666a-4652-9c73-b7021c90a5e7-kube-api-access-c7csb\" (UniqueName: \"kubernetes.io/projected/8f07b490-666a-4652-9c73-b7021c90a5e7-kube-api-access-c7csb\") pod \"8f07b490-666a-4652-9c73-b7021c90a5e7\" (UID: \"8f07b490-666a-4652-9c73-b7021c90a5e7\") " Nov 02 22:25:17.116485 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.116457 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f07b490-666a-4652-9c73-b7021c90a5e7-plugins-config-volume" pod "8f07b490-666a-4652-9c73-b7021c90a5e7" (UID: "8f07b490-666a-4652-9c73-b7021c90a5e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:17.116928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.116902 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f07b490-666a-4652-9c73-b7021c90a5e7-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:17.118399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.118372 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f07b490-666a-4652-9c73-b7021c90a5e7-kube-api-access-c7csb" pod "8f07b490-666a-4652-9c73-b7021c90a5e7" (UID: "8f07b490-666a-4652-9c73-b7021c90a5e7"). InnerVolumeSpecName "kube-api-access-c7csb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:17.218836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.218798 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7csb\" (UniqueName: \"kubernetes.io/projected/8f07b490-666a-4652-9c73-b7021c90a5e7-kube-api-access-c7csb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:17.268014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.267982 4309 generic.go:356] "Generic (PLEG): container finished" podID="8f07b490-666a-4652-9c73-b7021c90a5e7" containerID="1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5" exitCode=0 Nov 02 22:25:17.268215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.268028 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" Nov 02 22:25:17.268215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.268062 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" event={"ID":"8f07b490-666a-4652-9c73-b7021c90a5e7","Type":"ContainerDied","Data":"1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5"} Nov 02 22:25:17.268215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.268139 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w" event={"ID":"8f07b490-666a-4652-9c73-b7021c90a5e7","Type":"ContainerDied","Data":"794f41940466e31e94b2bf3df822d9c4f9a2955f1c44ef4aab6bf0e3128ae991"} Nov 02 22:25:17.268215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.268167 4309 scope.go:122] "RemoveContainer" containerID="1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5" Nov 02 22:25:17.270383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.270331 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" event={"ID":"15815a29-6f09-487a-88e6-3014b2361ee1","Type":"ContainerDied","Data":"502369d30ed796c3aaea5b573e4e8ba3c409907bb91911e1f1161de05fbef74c"} Nov 02 22:25:17.270383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.270377 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="502369d30ed796c3aaea5b573e4e8ba3c409907bb91911e1f1161de05fbef74c" Nov 02 22:25:17.270596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.270403 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr" Nov 02 22:25:17.290963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.282866 4309 scope.go:122] "RemoveContainer" containerID="1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5" Nov 02 22:25:17.290963 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:17.283262 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5\": container with ID starting with 1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5 not found: ID does not exist" containerID="1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5" Nov 02 22:25:17.290963 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.283292 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5"} err="failed to get container status \"1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5\": rpc error: code = NotFound desc = could not find container \"1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5\": container with ID starting with 1df62735df6e70f41d4cb6e04cd15d3713fa33349513429b167c126f783219d5 not found: ID does not exist" Nov 02 22:25:17.298529 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.297860 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w"] Nov 02 22:25:17.304749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:17.304719 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e49edd-76d7fb6b89-69s7w"] Nov 02 22:25:18.196035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.195960 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f07b490-666a-4652-9c73-b7021c90a5e7" path="/var/lib/kubelet/pods/8f07b490-666a-4652-9c73-b7021c90a5e7/volumes" Nov 02 22:25:18.351698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.346065 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg"] Nov 02 22:25:18.351698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.347460 4309 cpu_manager.go:414] "containerMap: removing container" podUID="8f07b490-666a-4652-9c73-b7021c90a5e7" containerName="payload-processing" Nov 02 22:25:18.351698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.347484 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f07b490-666a-4652-9c73-b7021c90a5e7" containerName="payload-processing" Nov 02 22:25:18.351698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.347508 4309 cpu_manager.go:414] "containerMap: removing container" podUID="15815a29-6f09-487a-88e6-3014b2361ee1" containerName="revoke-keys" Nov 02 22:25:18.351698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.347516 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15815a29-6f09-487a-88e6-3014b2361ee1" containerName="revoke-keys" Nov 02 22:25:18.351698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.347763 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f07b490-666a-4652-9c73-b7021c90a5e7" containerName="payload-processing" Nov 02 22:25:18.351698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.347786 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="15815a29-6f09-487a-88e6-3014b2361ee1" containerName="revoke-keys" Nov 02 22:25:18.421315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.421278 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg"] Nov 02 22:25:18.421491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.421409 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.426914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.424503 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:18.426914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.426001 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:25:18.426914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.426814 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:25:18.427178 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.426919 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:25:18.454979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.454722 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a82826b6-szkhg\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.455146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.455032 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8w4h\" (UniqueName: \"kubernetes.io/projected/edace62b-0f08-47f4-b6d4-83b0952e8242-kube-api-access-j8w4h\") pod \"maas-api-revoke-sub-a82826b6-szkhg\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.455146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.455096 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a82826b6-szkhg\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.556872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.556825 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a82826b6-szkhg\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.557051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.556984 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a82826b6-szkhg\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.557454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.557427 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8w4h\" (UniqueName: \"kubernetes.io/projected/edace62b-0f08-47f4-b6d4-83b0952e8242-kube-api-access-j8w4h\") pod \"maas-api-revoke-sub-a82826b6-szkhg\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.558099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.558076 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a82826b6-szkhg\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.559626 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.559601 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a82826b6-szkhg\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.565776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.565754 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8w4h\" (UniqueName: \"kubernetes.io/projected/edace62b-0f08-47f4-b6d4-83b0952e8242-kube-api-access-j8w4h\") pod \"maas-api-revoke-sub-a82826b6-szkhg\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:18.749140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:18.749059 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:19.104609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:19.096949 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg"] Nov 02 22:25:19.127236 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:19.127091 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedace62b_0f08_47f4_b6d4_83b0952e8242.slice/crio-47f180a3f5578e71c1d92547d70f00706122941ff51b3887e265fe20a9cd53b5 WatchSource:0}: Error finding container 47f180a3f5578e71c1d92547d70f00706122941ff51b3887e265fe20a9cd53b5: Status 404 returned error can't find the container with id 47f180a3f5578e71c1d92547d70f00706122941ff51b3887e265fe20a9cd53b5 Nov 02 22:25:19.311010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:19.310974 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" event={"ID":"edace62b-0f08-47f4-b6d4-83b0952e8242","Type":"ContainerStarted","Data":"47f180a3f5578e71c1d92547d70f00706122941ff51b3887e265fe20a9cd53b5"} Nov 02 22:25:20.331940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:20.331905 4309 generic.go:356] "Generic (PLEG): container finished" podID="edace62b-0f08-47f4-b6d4-83b0952e8242" containerID="f6a34f2d04fd4ef688d1c2a4aead899b66f783568558981ead62c10718b80311" exitCode=0 Nov 02 22:25:20.332345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:20.331992 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" event={"ID":"edace62b-0f08-47f4-b6d4-83b0952e8242","Type":"ContainerDied","Data":"f6a34f2d04fd4ef688d1c2a4aead899b66f783568558981ead62c10718b80311"} Nov 02 22:25:20.572568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:20.572511 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc"] Nov 02 22:25:20.572851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:20.572776 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" containerID="cri-o://5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f" gracePeriod=30 Nov 02 22:25:20.586845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:20.586764 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z"] Nov 02 22:25:20.587028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:20.586998 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" containerID="cri-o://f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7" gracePeriod=30 Nov 02 22:25:20.587490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:20.587433 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l"] Nov 02 22:25:20.587634 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:20.587609 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" containerID="cri-o://ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed" gracePeriod=30 Nov 02 22:25:20.776701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:20.769737 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" podUID="6e97fdbf-c24b-4727-b895-18915dc6b1a0" containerName="payload-pre-processing" containerID="cri-o://a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a" gracePeriod=25 Nov 02 22:25:21.041648 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.041617 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf"] Nov 02 22:25:21.059163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.059129 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf"] Nov 02 22:25:21.059342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.059248 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.072858 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:21.072827 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aab6102_cc3f_450a_8b3a_4656d2e4bbf2.slice/crio-conmon-d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:25:21.102922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.089949 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.102922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.090194 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2ss\" (UniqueName: \"kubernetes.io/projected/6ef72440-4080-4344-a5b1-236c6b4b84c0-kube-api-access-gw2ss\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.102922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.090262 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.191956 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.191923 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.192116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.192041 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2ss\" (UniqueName: \"kubernetes.io/projected/6ef72440-4080-4344-a5b1-236c6b4b84c0-kube-api-access-gw2ss\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.192116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.192069 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.192718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.192690 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.194372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.194347 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.220733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.207075 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2ss\" (UniqueName: \"kubernetes.io/projected/6ef72440-4080-4344-a5b1-236c6b4b84c0-kube-api-access-gw2ss\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.290731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.290698 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:25:21.347146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.347107 4309 generic.go:356] "Generic (PLEG): container finished" podID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerID="d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3" exitCode=0 Nov 02 22:25:21.347146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.347150 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" Nov 02 22:25:21.347594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.347185 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" event={"ID":"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2","Type":"ContainerDied","Data":"d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3"} Nov 02 22:25:21.347594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.347242 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6" event={"ID":"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2","Type":"ContainerDied","Data":"47eabb83794617f867f6522eaf8a47e1f65a0901aeda296bc90036ddf5264ce1"} Nov 02 22:25:21.347594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.347299 4309 scope.go:122] "RemoveContainer" containerID="d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3" Nov 02 22:25:21.367229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.367207 4309 scope.go:122] "RemoveContainer" containerID="d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3" Nov 02 22:25:21.370819 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:21.370783 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3\": container with ID starting with d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3 not found: ID does not exist" containerID="d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3" Nov 02 22:25:21.370957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.370830 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3"} err="failed to get container status \"d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3\": rpc error: code = NotFound desc = could not find container \"d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3\": container with ID starting with d4a83e180be4e912eb5e4ac8122e69a0a55e1edd68c5322b230dfa1b60cda9c3 not found: ID does not exist" Nov 02 22:25:21.393989 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.393953 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istiod-ca-cert\") pod \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " Nov 02 22:25:21.394161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394037 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-envoy\") pod \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " Nov 02 22:25:21.394234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394209 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-socket\") pod \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " Nov 02 22:25:21.394301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394267 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istiod-ca-cert" pod "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" (UID: "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:21.394301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394275 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-certs\") pod \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " Nov 02 22:25:21.394408 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394340 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-podinfo\") pod \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " Nov 02 22:25:21.394488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394469 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-certs" pod "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" (UID: "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:21.394552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394519 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-kube-api-access-zhw7b\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-kube-api-access-zhw7b\") pod \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " Nov 02 22:25:21.394610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394570 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-token\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-token\") pod \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " Nov 02 22:25:21.394610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394596 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-data\") pod \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " Nov 02 22:25:21.394740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394624 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-credential-socket\") pod \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\" (UID: \"5aab6102-cc3f-450a-8b3a-4656d2e4bbf2\") " Nov 02 22:25:21.394740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.394641 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-socket" pod "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" (UID: "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:21.395288 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.395262 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-data" pod "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" (UID: "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:21.395398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.395317 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.395398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.395339 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.395398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.395353 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.395563 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.395481 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-credential-socket" pod "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" (UID: "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:21.396875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.396841 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-envoy" pod "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" (UID: "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:21.397110 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.397085 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-kube-api-access-zhw7b" pod "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" (UID: "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2"). InnerVolumeSpecName "kube-api-access-zhw7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:21.397324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.397305 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-podinfo" pod "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" (UID: "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:21.397461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.397442 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-token" pod "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" (UID: "5aab6102-cc3f-450a-8b3a-4656d2e4bbf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:21.401661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.401636 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:21.497568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.497527 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhw7b\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-kube-api-access-zhw7b\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.497812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.497577 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.497812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.497594 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.497812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.497608 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.497812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.497623 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.497812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.497636 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.560391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.560363 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:21.599696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.599311 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-openshift-ca\") pod \"edace62b-0f08-47f4-b6d4-83b0952e8242\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " Nov 02 22:25:21.599696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.599442 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-xks-ca\") pod \"edace62b-0f08-47f4-b6d4-83b0952e8242\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " Nov 02 22:25:21.599696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.599599 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edace62b-0f08-47f4-b6d4-83b0952e8242-kube-api-access-j8w4h\" (UniqueName: \"kubernetes.io/projected/edace62b-0f08-47f4-b6d4-83b0952e8242-kube-api-access-j8w4h\") pod \"edace62b-0f08-47f4-b6d4-83b0952e8242\" (UID: \"edace62b-0f08-47f4-b6d4-83b0952e8242\") " Nov 02 22:25:21.602322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.602035 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-openshift-ca" pod "edace62b-0f08-47f4-b6d4-83b0952e8242" (UID: "edace62b-0f08-47f4-b6d4-83b0952e8242"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:21.618198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.618156 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edace62b-0f08-47f4-b6d4-83b0952e8242-kube-api-access-j8w4h" pod "edace62b-0f08-47f4-b6d4-83b0952e8242" (UID: "edace62b-0f08-47f4-b6d4-83b0952e8242"). InnerVolumeSpecName "kube-api-access-j8w4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:21.618328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.618311 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-xks-ca" pod "edace62b-0f08-47f4-b6d4-83b0952e8242" (UID: "edace62b-0f08-47f4-b6d4-83b0952e8242"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:21.680004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.679969 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6"] Nov 02 22:25:21.684790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.684755 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-r27p6"] Nov 02 22:25:21.702585 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.702552 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.702585 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.702586 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8w4h\" (UniqueName: \"kubernetes.io/projected/edace62b-0f08-47f4-b6d4-83b0952e8242-kube-api-access-j8w4h\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.702798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.702599 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edace62b-0f08-47f4-b6d4-83b0952e8242-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:21.715128 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:21.715082 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef72440_4080_4344_a5b1_236c6b4b84c0.slice/crio-cb091577e413ddd5b37ef346faa904cf975ff792b21651acd4c95361eab7b1d2 WatchSource:0}: Error finding container cb091577e413ddd5b37ef346faa904cf975ff792b21651acd4c95361eab7b1d2: Status 404 returned error can't find the container with id cb091577e413ddd5b37ef346faa904cf975ff792b21651acd4c95361eab7b1d2 Nov 02 22:25:21.716069 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:21.716042 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf"] Nov 02 22:25:22.193378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:22.193348 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" path="/var/lib/kubelet/pods/5aab6102-cc3f-450a-8b3a-4656d2e4bbf2/volumes" Nov 02 22:25:22.356042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:22.355966 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" Nov 02 22:25:22.356042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:22.356003 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg" event={"ID":"edace62b-0f08-47f4-b6d4-83b0952e8242","Type":"ContainerDied","Data":"47f180a3f5578e71c1d92547d70f00706122941ff51b3887e265fe20a9cd53b5"} Nov 02 22:25:22.356542 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:22.356051 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47f180a3f5578e71c1d92547d70f00706122941ff51b3887e265fe20a9cd53b5" Nov 02 22:25:22.359484 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:22.359457 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" event={"ID":"6ef72440-4080-4344-a5b1-236c6b4b84c0","Type":"ContainerStarted","Data":"e892a5da4173b8b4a3f0a21ca03240779b28399da815c17cfbb0e48a78009d86"} Nov 02 22:25:22.359600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:22.359510 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" event={"ID":"6ef72440-4080-4344-a5b1-236c6b4b84c0","Type":"ContainerStarted","Data":"cb091577e413ddd5b37ef346faa904cf975ff792b21651acd4c95361eab7b1d2"} Nov 02 22:25:22.389366 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:22.389316 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" podStartSLOduration=1.389299654 podStartE2EDuration="1.389299654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:22.380514797 +0000 UTC m=+1650.318628018" watchObservedRunningTime="2026-11-02 22:25:22.389299654 +0000 UTC m=+1650.327412873" Nov 02 22:25:23.165190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:23.165157 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn"] Nov 02 22:25:23.178100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:23.178066 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w"] Nov 02 22:25:23.370828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:23.370789 4309 generic.go:356] "Generic (PLEG): container finished" podID="6ef72440-4080-4344-a5b1-236c6b4b84c0" containerID="e892a5da4173b8b4a3f0a21ca03240779b28399da815c17cfbb0e48a78009d86" exitCode=0 Nov 02 22:25:23.371233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:23.370869 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" event={"ID":"6ef72440-4080-4344-a5b1-236c6b4b84c0","Type":"ContainerDied","Data":"e892a5da4173b8b4a3f0a21ca03240779b28399da815c17cfbb0e48a78009d86"} Nov 02 22:25:24.557865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.557839 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:24.699707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.699612 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef72440-4080-4344-a5b1-236c6b4b84c0-kube-api-access-gw2ss\" (UniqueName: \"kubernetes.io/projected/6ef72440-4080-4344-a5b1-236c6b4b84c0-kube-api-access-gw2ss\") pod \"6ef72440-4080-4344-a5b1-236c6b4b84c0\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " Nov 02 22:25:24.699882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.699738 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-openshift-ca\") pod \"6ef72440-4080-4344-a5b1-236c6b4b84c0\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " Nov 02 22:25:24.699882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.699780 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-xks-ca\") pod \"6ef72440-4080-4344-a5b1-236c6b4b84c0\" (UID: \"6ef72440-4080-4344-a5b1-236c6b4b84c0\") " Nov 02 22:25:24.700180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.700152 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-openshift-ca" pod "6ef72440-4080-4344-a5b1-236c6b4b84c0" (UID: "6ef72440-4080-4344-a5b1-236c6b4b84c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:24.700530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.700508 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:24.702034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.702010 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-xks-ca" pod "6ef72440-4080-4344-a5b1-236c6b4b84c0" (UID: "6ef72440-4080-4344-a5b1-236c6b4b84c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:24.702170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.702106 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef72440-4080-4344-a5b1-236c6b4b84c0-kube-api-access-gw2ss" pod "6ef72440-4080-4344-a5b1-236c6b4b84c0" (UID: "6ef72440-4080-4344-a5b1-236c6b4b84c0"). InnerVolumeSpecName "kube-api-access-gw2ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:24.812696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.807098 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw2ss\" (UniqueName: \"kubernetes.io/projected/6ef72440-4080-4344-a5b1-236c6b4b84c0-kube-api-access-gw2ss\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:24.812696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.807140 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef72440-4080-4344-a5b1-236c6b4b84c0-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:24.869952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:24.869917 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" podUID="6e97fdbf-c24b-4727-b895-18915dc6b1a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:25:25.275299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:25.275259 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" podUID="38542064-c103-49d1-9ce6-f0fd1398ac97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:25:25.388820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:25.388793 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" Nov 02 22:25:25.388977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:25.388832 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf" event={"ID":"6ef72440-4080-4344-a5b1-236c6b4b84c0","Type":"ContainerDied","Data":"cb091577e413ddd5b37ef346faa904cf975ff792b21651acd4c95361eab7b1d2"} Nov 02 22:25:25.388977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:25.388883 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb091577e413ddd5b37ef346faa904cf975ff792b21651acd4c95361eab7b1d2" Nov 02 22:25:25.910848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:25.910818 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:25:25.918333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:25.918310 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:25:25.990964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:25.990936 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:25:26.026782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.026752 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-envoy\") pod \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " Nov 02 22:25:26.026966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.026797 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-envoy\") pod \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " Nov 02 22:25:26.026966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.026833 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-kube-api-access-j7w9j\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-kube-api-access-j7w9j\") pod \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " Nov 02 22:25:26.026966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.026868 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-socket\") pod \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " Nov 02 22:25:26.026966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.026915 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-data\") pod \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " Nov 02 22:25:26.027182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.026994 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-token\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-token\") pod \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " Nov 02 22:25:26.027182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027042 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-kube-api-access-dq479\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-kube-api-access-dq479\") pod \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " Nov 02 22:25:26.027182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027078 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-credential-socket\") pod \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " Nov 02 22:25:26.027335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027210 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istiod-ca-cert\") pod \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " Nov 02 22:25:26.027335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027250 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-socket" pod "dc4b5c51-e840-421e-8353-f1f3b7a604f1" (UID: "dc4b5c51-e840-421e-8353-f1f3b7a604f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.027335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027259 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-podinfo\") pod \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " Nov 02 22:25:26.027335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027302 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-certs\") pod \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " Nov 02 22:25:26.027538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027333 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-socket\") pod \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " Nov 02 22:25:26.027538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027373 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-token\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-token\") pod \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " Nov 02 22:25:26.027538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027416 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-credential-socket\") pod \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " Nov 02 22:25:26.027538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027461 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-data\") pod \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " Nov 02 22:25:26.027784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027571 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-podinfo\") pod \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " Nov 02 22:25:26.027784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027668 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istiod-ca-cert\") pod \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\" (UID: \"dc4b5c51-e840-421e-8353-f1f3b7a604f1\") " Nov 02 22:25:26.027784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.027744 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-certs\") pod \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\" (UID: \"4b47633d-d531-4e0c-a9ab-2942027ae6cd\") " Nov 02 22:25:26.028612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.028580 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.028861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.028841 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-certs" pod "4b47633d-d531-4e0c-a9ab-2942027ae6cd" (UID: "4b47633d-d531-4e0c-a9ab-2942027ae6cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.029122 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.029098 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-data" pod "dc4b5c51-e840-421e-8353-f1f3b7a604f1" (UID: "dc4b5c51-e840-421e-8353-f1f3b7a604f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.029378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.029360 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-envoy" pod "4b47633d-d531-4e0c-a9ab-2942027ae6cd" (UID: "4b47633d-d531-4e0c-a9ab-2942027ae6cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.029968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.029872 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-data" pod "4b47633d-d531-4e0c-a9ab-2942027ae6cd" (UID: "4b47633d-d531-4e0c-a9ab-2942027ae6cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.029968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.029954 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-envoy" pod "dc4b5c51-e840-421e-8353-f1f3b7a604f1" (UID: "dc4b5c51-e840-421e-8353-f1f3b7a604f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.030135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.030000 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-credential-socket" pod "dc4b5c51-e840-421e-8353-f1f3b7a604f1" (UID: "dc4b5c51-e840-421e-8353-f1f3b7a604f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.030351 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.030235 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istiod-ca-cert" pod "4b47633d-d531-4e0c-a9ab-2942027ae6cd" (UID: "4b47633d-d531-4e0c-a9ab-2942027ae6cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:26.030805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.030562 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istiod-ca-cert" pod "dc4b5c51-e840-421e-8353-f1f3b7a604f1" (UID: "dc4b5c51-e840-421e-8353-f1f3b7a604f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:26.031108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.030944 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-certs" pod "dc4b5c51-e840-421e-8353-f1f3b7a604f1" (UID: "dc4b5c51-e840-421e-8353-f1f3b7a604f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.032308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.031292 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-credential-socket" pod "4b47633d-d531-4e0c-a9ab-2942027ae6cd" (UID: "4b47633d-d531-4e0c-a9ab-2942027ae6cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.032308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.031563 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-token" pod "dc4b5c51-e840-421e-8353-f1f3b7a604f1" (UID: "dc4b5c51-e840-421e-8353-f1f3b7a604f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:26.032308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.032016 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-token" pod "4b47633d-d531-4e0c-a9ab-2942027ae6cd" (UID: "4b47633d-d531-4e0c-a9ab-2942027ae6cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:26.032708 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.032685 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-socket" pod "4b47633d-d531-4e0c-a9ab-2942027ae6cd" (UID: "4b47633d-d531-4e0c-a9ab-2942027ae6cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.033104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.033078 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-kube-api-access-j7w9j" pod "dc4b5c51-e840-421e-8353-f1f3b7a604f1" (UID: "dc4b5c51-e840-421e-8353-f1f3b7a604f1"). InnerVolumeSpecName "kube-api-access-j7w9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:26.033741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.033708 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-podinfo" pod "dc4b5c51-e840-421e-8353-f1f3b7a604f1" (UID: "dc4b5c51-e840-421e-8353-f1f3b7a604f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:26.033862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.033831 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-podinfo" pod "4b47633d-d531-4e0c-a9ab-2942027ae6cd" (UID: "4b47633d-d531-4e0c-a9ab-2942027ae6cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:26.034549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.034527 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-kube-api-access-dq479" pod "4b47633d-d531-4e0c-a9ab-2942027ae6cd" (UID: "4b47633d-d531-4e0c-a9ab-2942027ae6cd"). InnerVolumeSpecName "kube-api-access-dq479". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:26.129426 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129342 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-socket\") pod \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " Nov 02 22:25:26.129582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129541 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-credential-socket\") pod \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " Nov 02 22:25:26.129629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129600 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-envoy\") pod \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " Nov 02 22:25:26.129629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129604 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-socket" pod "4b69bd1d-28c3-4c79-8a0f-4442d80b1691" (UID: "4b69bd1d-28c3-4c79-8a0f-4442d80b1691"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.129759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129690 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-podinfo\") pod \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " Nov 02 22:25:26.129759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129753 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-certs\") pod \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " Nov 02 22:25:26.129882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129853 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-token\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-token\") pod \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " Nov 02 22:25:26.129945 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129882 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-credential-socket" pod "4b69bd1d-28c3-4c79-8a0f-4442d80b1691" (UID: "4b69bd1d-28c3-4c79-8a0f-4442d80b1691"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.129945 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129907 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istiod-ca-cert\") pod \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " Nov 02 22:25:26.130085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129944 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-kube-api-access-mf5f2\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-kube-api-access-mf5f2\") pod \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " Nov 02 22:25:26.130085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.129992 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-certs" pod "4b69bd1d-28c3-4c79-8a0f-4442d80b1691" (UID: "4b69bd1d-28c3-4c79-8a0f-4442d80b1691"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.130200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130103 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-data\") pod \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\" (UID: \"4b69bd1d-28c3-4c79-8a0f-4442d80b1691\") " Nov 02 22:25:26.130355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130308 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istiod-ca-cert" pod "4b69bd1d-28c3-4c79-8a0f-4442d80b1691" (UID: "4b69bd1d-28c3-4c79-8a0f-4442d80b1691"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:26.130558 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130538 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-data" pod "4b69bd1d-28c3-4c79-8a0f-4442d80b1691" (UID: "4b69bd1d-28c3-4c79-8a0f-4442d80b1691"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.130905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130883 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130910 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130926 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7w9j\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-kube-api-access-j7w9j\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130939 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130952 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130965 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130978 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq479\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-kube-api-access-dq479\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.130991 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131018 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131033 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131050 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131065 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131080 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131094 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131106 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131118 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4b5c51-e840-421e-8353-f1f3b7a604f1-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131130 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131145 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131158 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131172 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131188 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4b5c51-e840-421e-8353-f1f3b7a604f1-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.131348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.131202 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b47633d-d531-4e0c-a9ab-2942027ae6cd-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.132329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.132306 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-token" pod "4b69bd1d-28c3-4c79-8a0f-4442d80b1691" (UID: "4b69bd1d-28c3-4c79-8a0f-4442d80b1691"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:26.132442 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.132401 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-podinfo" pod "4b69bd1d-28c3-4c79-8a0f-4442d80b1691" (UID: "4b69bd1d-28c3-4c79-8a0f-4442d80b1691"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:26.132649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.132622 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-kube-api-access-mf5f2" pod "4b69bd1d-28c3-4c79-8a0f-4442d80b1691" (UID: "4b69bd1d-28c3-4c79-8a0f-4442d80b1691"). InnerVolumeSpecName "kube-api-access-mf5f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:26.133699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.133658 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-envoy" pod "4b69bd1d-28c3-4c79-8a0f-4442d80b1691" (UID: "4b69bd1d-28c3-4c79-8a0f-4442d80b1691"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:26.250704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.249500 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.250704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.249547 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.250704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.249562 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.250704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.249577 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf5f2\" (UniqueName: \"kubernetes.io/projected/4b69bd1d-28c3-4c79-8a0f-4442d80b1691-kube-api-access-mf5f2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:26.398397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.398316 4309 generic.go:356] "Generic (PLEG): container finished" podID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerID="ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed" exitCode=0 Nov 02 22:25:26.398397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.398358 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" Nov 02 22:25:26.398616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.398395 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" event={"ID":"4b69bd1d-28c3-4c79-8a0f-4442d80b1691","Type":"ContainerDied","Data":"ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed"} Nov 02 22:25:26.398616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.398444 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l" event={"ID":"4b69bd1d-28c3-4c79-8a0f-4442d80b1691","Type":"ContainerDied","Data":"504e887202d24208b504dc8ea206f91f262c9363882bcd445a6d829c6d2e114b"} Nov 02 22:25:26.398616 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.398465 4309 scope.go:122] "RemoveContainer" containerID="ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed" Nov 02 22:25:26.400481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.400457 4309 generic.go:356] "Generic (PLEG): container finished" podID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerID="f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7" exitCode=0 Nov 02 22:25:26.400630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.400518 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" Nov 02 22:25:26.400630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.400555 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" event={"ID":"4b47633d-d531-4e0c-a9ab-2942027ae6cd","Type":"ContainerDied","Data":"f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7"} Nov 02 22:25:26.400630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.400595 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z" event={"ID":"4b47633d-d531-4e0c-a9ab-2942027ae6cd","Type":"ContainerDied","Data":"3b66deb99eee4740219acd334531a417734a15bcdaf04078418a90d8bbb69c8f"} Nov 02 22:25:26.402617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.402588 4309 generic.go:356] "Generic (PLEG): container finished" podID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerID="5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f" exitCode=0 Nov 02 22:25:26.402760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.402640 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" Nov 02 22:25:26.402825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.402777 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" event={"ID":"dc4b5c51-e840-421e-8353-f1f3b7a604f1","Type":"ContainerDied","Data":"5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f"} Nov 02 22:25:26.402898 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.402846 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc" event={"ID":"dc4b5c51-e840-421e-8353-f1f3b7a604f1","Type":"ContainerDied","Data":"8127adb0467cd1d524ac3f36b7c1c30891d62154699fc0e6ae23958dc4861cad"} Nov 02 22:25:26.412239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.412222 4309 scope.go:122] "RemoveContainer" containerID="ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed" Nov 02 22:25:26.412530 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:26.412508 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed\": container with ID starting with ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed not found: ID does not exist" containerID="ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed" Nov 02 22:25:26.412600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.412541 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed"} err="failed to get container status \"ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed\": rpc error: code = NotFound desc = could not find container \"ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed\": container with ID starting with ce63042f5ccd553f6ef0c8973a4767232a4e2d0d6338c6c236af643736b434ed not found: ID does not exist" Nov 02 22:25:26.412600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.412569 4309 scope.go:122] "RemoveContainer" containerID="f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7" Nov 02 22:25:26.425356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.425328 4309 scope.go:122] "RemoveContainer" containerID="4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d" Nov 02 22:25:26.426461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.426408 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z"] Nov 02 22:25:26.431202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.431183 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-9672z"] Nov 02 22:25:26.437063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.437047 4309 scope.go:122] "RemoveContainer" containerID="f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7" Nov 02 22:25:26.437361 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:26.437344 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7\": container with ID starting with f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7 not found: ID does not exist" containerID="f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7" Nov 02 22:25:26.437439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.437367 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7"} err="failed to get container status \"f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7\": rpc error: code = NotFound desc = could not find container \"f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7\": container with ID starting with f0ef13ea1f0bf7a1ed61a47f78e78f5595c2813be837f023124c828df855cfe7 not found: ID does not exist" Nov 02 22:25:26.437439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.437383 4309 scope.go:122] "RemoveContainer" containerID="4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d" Nov 02 22:25:26.437649 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:26.437628 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d\": container with ID starting with 4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d not found: ID does not exist" containerID="4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d" Nov 02 22:25:26.437866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.437655 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d"} err="failed to get container status \"4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d\": rpc error: code = NotFound desc = could not find container \"4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d\": container with ID starting with 4acd359fc393ce9995565768e2f1ea19e473186cfbbc7d07982300a9cd495d4d not found: ID does not exist" Nov 02 22:25:26.437866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.437703 4309 scope.go:122] "RemoveContainer" containerID="5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f" Nov 02 22:25:26.440612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.440590 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc"] Nov 02 22:25:26.444850 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.444828 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-f6gsc"] Nov 02 22:25:26.460831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.460805 4309 scope.go:122] "RemoveContainer" containerID="5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f" Nov 02 22:25:26.461223 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:26.461203 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f\": container with ID starting with 5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f not found: ID does not exist" containerID="5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f" Nov 02 22:25:26.461331 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.461229 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f"} err="failed to get container status \"5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f\": rpc error: code = NotFound desc = could not find container \"5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f\": container with ID starting with 5b8b66db94a9127dbbe0f839f9b5d129edb34db424a2bd22b79199be5dfaa13f not found: ID does not exist" Nov 02 22:25:26.461331 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.461279 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l"] Nov 02 22:25:26.467874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:26.467853 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e49edd-openshift-default-8486ddd84c-q9z8l"] Nov 02 22:25:27.921507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:27.921473 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp"] Nov 02 22:25:27.927119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:27.927091 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e30968-ml8sp"] Nov 02 22:25:28.165438 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.165402 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" podUID="e7a03376-19f0-4899-ad29-66280272e7d6" containerName="payload-processing" containerID="cri-o://3283cc8a8fe5792f4a2270c2222b4eb03d371b31401e5332529283193ec22b95" gracePeriod=25 Nov 02 22:25:28.179955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.179873 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" podUID="ffc1bd6d-fcec-4658-9968-4bd9bd401d15" containerName="payload-pre-processing" containerID="cri-o://7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d" gracePeriod=25 Nov 02 22:25:28.191665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.191642 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" path="/var/lib/kubelet/pods/4b47633d-d531-4e0c-a9ab-2942027ae6cd/volumes" Nov 02 22:25:28.192263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.192246 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" path="/var/lib/kubelet/pods/4b69bd1d-28c3-4c79-8a0f-4442d80b1691/volumes" Nov 02 22:25:28.192750 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.192736 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" path="/var/lib/kubelet/pods/dc4b5c51-e840-421e-8353-f1f3b7a604f1/volumes" Nov 02 22:25:28.193480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.193465 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb3d6ea-9ee0-404d-a41a-ed026d70a903" path="/var/lib/kubelet/pods/ebb3d6ea-9ee0-404d-a41a-ed026d70a903/volumes" Nov 02 22:25:28.456742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.456709 4309 generic.go:356] "Generic (PLEG): container finished" podID="e7a03376-19f0-4899-ad29-66280272e7d6" containerID="3283cc8a8fe5792f4a2270c2222b4eb03d371b31401e5332529283193ec22b95" exitCode=0 Nov 02 22:25:28.456867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.456753 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" event={"ID":"e7a03376-19f0-4899-ad29-66280272e7d6","Type":"ContainerDied","Data":"3283cc8a8fe5792f4a2270c2222b4eb03d371b31401e5332529283193ec22b95"} Nov 02 22:25:28.492851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.492818 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:25:28.545887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.545850 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a03376-19f0-4899-ad29-66280272e7d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a03376-19f0-4899-ad29-66280272e7d6-plugins-config-volume\") pod \"e7a03376-19f0-4899-ad29-66280272e7d6\" (UID: \"e7a03376-19f0-4899-ad29-66280272e7d6\") " Nov 02 22:25:28.546098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.546078 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a03376-19f0-4899-ad29-66280272e7d6-kube-api-access-rqz5r\" (UniqueName: \"kubernetes.io/projected/e7a03376-19f0-4899-ad29-66280272e7d6-kube-api-access-rqz5r\") pod \"e7a03376-19f0-4899-ad29-66280272e7d6\" (UID: \"e7a03376-19f0-4899-ad29-66280272e7d6\") " Nov 02 22:25:28.546358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.546318 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a03376-19f0-4899-ad29-66280272e7d6-plugins-config-volume" pod "e7a03376-19f0-4899-ad29-66280272e7d6" (UID: "e7a03376-19f0-4899-ad29-66280272e7d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:28.546854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.546833 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a03376-19f0-4899-ad29-66280272e7d6-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:28.548236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.548209 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a03376-19f0-4899-ad29-66280272e7d6-kube-api-access-rqz5r" pod "e7a03376-19f0-4899-ad29-66280272e7d6" (UID: "e7a03376-19f0-4899-ad29-66280272e7d6"). InnerVolumeSpecName "kube-api-access-rqz5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:28.648671 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:28.648639 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqz5r\" (UniqueName: \"kubernetes.io/projected/e7a03376-19f0-4899-ad29-66280272e7d6-kube-api-access-rqz5r\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:29.330956 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.330922 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm"] Nov 02 22:25:29.331742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331719 4309 cpu_manager.go:414] "containerMap: removing container" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" Nov 02 22:25:29.331861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331744 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" Nov 02 22:25:29.331861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331764 4309 cpu_manager.go:414] "containerMap: removing container" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" Nov 02 22:25:29.331861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331771 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" Nov 02 22:25:29.331861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331791 4309 cpu_manager.go:414] "containerMap: removing container" podUID="6ef72440-4080-4344-a5b1-236c6b4b84c0" containerName="revoke-keys" Nov 02 22:25:29.331861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331799 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef72440-4080-4344-a5b1-236c6b4b84c0" containerName="revoke-keys" Nov 02 22:25:29.331861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331821 4309 cpu_manager.go:414] "containerMap: removing container" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" Nov 02 22:25:29.331861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331828 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" Nov 02 22:25:29.331861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331849 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e7a03376-19f0-4899-ad29-66280272e7d6" containerName="payload-processing" Nov 02 22:25:29.331861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331855 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a03376-19f0-4899-ad29-66280272e7d6" containerName="payload-processing" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331868 4309 cpu_manager.go:414] "containerMap: removing container" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331874 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331891 4309 cpu_manager.go:414] "containerMap: removing container" podUID="edace62b-0f08-47f4-b6d4-83b0952e8242" containerName="revoke-keys" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331897 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edace62b-0f08-47f4-b6d4-83b0952e8242" containerName="revoke-keys" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331949 4309 cpu_manager.go:414] "containerMap: removing container" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.331967 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.332190 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="edace62b-0f08-47f4-b6d4-83b0952e8242" containerName="revoke-keys" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.332212 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4b5c51-e840-421e-8353-f1f3b7a604f1" containerName="istio-proxy" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.332228 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aab6102-cc3f-450a-8b3a-4656d2e4bbf2" containerName="istio-proxy" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.332236 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b69bd1d-28c3-4c79-8a0f-4442d80b1691" containerName="istio-proxy" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.332247 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef72440-4080-4344-a5b1-236c6b4b84c0" containerName="revoke-keys" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.332259 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.332270 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a03376-19f0-4899-ad29-66280272e7d6" containerName="payload-processing" Nov 02 22:25:29.332307 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.332281 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b47633d-d531-4e0c-a9ab-2942027ae6cd" containerName="istio-proxy" Nov 02 22:25:29.336544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.336512 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.341410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.341386 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:29.341576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.341426 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:25:29.343358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.343333 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm"] Nov 02 22:25:29.376956 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.376928 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:25:29.377132 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.376941 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:25:29.465822 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.465789 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.466039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.465867 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f7g4\" (UniqueName: \"kubernetes.io/projected/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-kube-api-access-4f7g4\") pod \"maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.466039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.466015 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.470052 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.470021 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" Nov 02 22:25:29.470052 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.470038 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn" event={"ID":"e7a03376-19f0-4899-ad29-66280272e7d6","Type":"ContainerDied","Data":"4a49a18cfb67026984c479868cd05b962cdaec6cc7afe27309e76c78b41938ef"} Nov 02 22:25:29.470280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.470116 4309 scope.go:122] "RemoveContainer" containerID="3283cc8a8fe5792f4a2270c2222b4eb03d371b31401e5332529283193ec22b95" Nov 02 22:25:29.499943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.499611 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn"] Nov 02 22:25:29.514070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.507368 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a54217-5c878f886f-z4cvn"] Nov 02 22:25:29.567413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.567376 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.567591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.567576 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.567722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.567664 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7g4\" (UniqueName: \"kubernetes.io/projected/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-kube-api-access-4f7g4\") pod \"maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.568379 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.568330 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.569952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.569934 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.575889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.575869 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f7g4\" (UniqueName: \"kubernetes.io/projected/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-kube-api-access-4f7g4\") pod \"maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.656782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.656700 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:29.942310 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:29.942280 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c26468_887a_4ac2_ba5b_2e9cc91ffe2e.slice/crio-c0f007a5d44dc4f06b2d913148a6ef064aa91ae6721c20a2ce91c58a8e793f8c WatchSource:0}: Error finding container c0f007a5d44dc4f06b2d913148a6ef064aa91ae6721c20a2ce91c58a8e793f8c: Status 404 returned error can't find the container with id c0f007a5d44dc4f06b2d913148a6ef064aa91ae6721c20a2ce91c58a8e793f8c Nov 02 22:25:29.942530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:29.942508 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm"] Nov 02 22:25:30.209776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:30.209742 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a03376-19f0-4899-ad29-66280272e7d6" path="/var/lib/kubelet/pods/e7a03376-19f0-4899-ad29-66280272e7d6/volumes" Nov 02 22:25:30.481235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:30.481154 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" event={"ID":"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e","Type":"ContainerStarted","Data":"732750424ae5d507b52c5d919817fd422eeb50a93a4e7f4e9d42d42d209d9c55"} Nov 02 22:25:30.481235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:30.481204 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" event={"ID":"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e","Type":"ContainerStarted","Data":"c0f007a5d44dc4f06b2d913148a6ef064aa91ae6721c20a2ce91c58a8e793f8c"} Nov 02 22:25:30.496901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:30.496846 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" podStartSLOduration=1.496827369 podStartE2EDuration="1.496827369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:30.495579355 +0000 UTC m=+1658.433692574" watchObservedRunningTime="2026-11-02 22:25:30.496827369 +0000 UTC m=+1658.434940587" Nov 02 22:25:31.232755 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:31.225649 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr"] Nov 02 22:25:31.241763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:31.236998 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a54217-fd7fr"] Nov 02 22:25:31.492226 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:31.492137 4309 generic.go:356] "Generic (PLEG): container finished" podID="30c26468-887a-4ac2-ba5b-2e9cc91ffe2e" containerID="732750424ae5d507b52c5d919817fd422eeb50a93a4e7f4e9d42d42d209d9c55" exitCode=0 Nov 02 22:25:31.492651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:31.492218 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" event={"ID":"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e","Type":"ContainerDied","Data":"732750424ae5d507b52c5d919817fd422eeb50a93a4e7f4e9d42d42d209d9c55"} Nov 02 22:25:32.203494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.203457 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15815a29-6f09-487a-88e6-3014b2361ee1" path="/var/lib/kubelet/pods/15815a29-6f09-487a-88e6-3014b2361ee1/volumes" Nov 02 22:25:32.671138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.671115 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:32.751973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.751884 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-kube-api-access-4f7g4\" (UniqueName: \"kubernetes.io/projected/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-kube-api-access-4f7g4\") pod \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " Nov 02 22:25:32.751973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.751957 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-openshift-ca\") pod \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " Nov 02 22:25:32.752206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.752035 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-xks-ca\") pod \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\" (UID: \"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e\") " Nov 02 22:25:32.752338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.752315 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-openshift-ca" pod "30c26468-887a-4ac2-ba5b-2e9cc91ffe2e" (UID: "30c26468-887a-4ac2-ba5b-2e9cc91ffe2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:32.752625 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.752605 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:32.754061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.754040 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-xks-ca" pod "30c26468-887a-4ac2-ba5b-2e9cc91ffe2e" (UID: "30c26468-887a-4ac2-ba5b-2e9cc91ffe2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:32.754121 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.754045 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-kube-api-access-4f7g4" pod "30c26468-887a-4ac2-ba5b-2e9cc91ffe2e" (UID: "30c26468-887a-4ac2-ba5b-2e9cc91ffe2e"). InnerVolumeSpecName "kube-api-access-4f7g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:32.863700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.858972 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f7g4\" (UniqueName: \"kubernetes.io/projected/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-kube-api-access-4f7g4\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:32.863700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:32.859011 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:33.512013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:33.511982 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" Nov 02 22:25:33.512204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:33.511978 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm" event={"ID":"30c26468-887a-4ac2-ba5b-2e9cc91ffe2e","Type":"ContainerDied","Data":"c0f007a5d44dc4f06b2d913148a6ef064aa91ae6721c20a2ce91c58a8e793f8c"} Nov 02 22:25:33.512204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:33.512112 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0f007a5d44dc4f06b2d913148a6ef064aa91ae6721c20a2ce91c58a8e793f8c" Nov 02 22:25:34.870832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:34.870787 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" podUID="6e97fdbf-c24b-4727-b895-18915dc6b1a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:25:35.055699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.055070 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" podUID="ffc1bd6d-fcec-4658-9968-4bd9bd401d15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:25:35.287255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.287025 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" podUID="38542064-c103-49d1-9ce6-f0fd1398ac97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:25:35.377805 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.377771 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb"] Nov 02 22:25:35.378494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.378480 4309 cpu_manager.go:414] "containerMap: removing container" podUID="30c26468-887a-4ac2-ba5b-2e9cc91ffe2e" containerName="revoke-keys" Nov 02 22:25:35.378538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.378496 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c26468-887a-4ac2-ba5b-2e9cc91ffe2e" containerName="revoke-keys" Nov 02 22:25:35.378611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.378602 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c26468-887a-4ac2-ba5b-2e9cc91ffe2e" containerName="revoke-keys" Nov 02 22:25:35.382438 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.382418 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.388409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.387621 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:25:35.388409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.388049 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:25:35.392503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.392481 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb"] Nov 02 22:25:35.473875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.473828 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:35.474027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.473940 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:25:35.537903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.537807 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00c72a-879tb\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.537903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.537879 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x4wp\" (UniqueName: \"kubernetes.io/projected/221a9e6f-803e-40f8-be5d-4db112c93e4b-kube-api-access-5x4wp\") pod \"maas-api-revoke-keys-e2e-inf-b-00c72a-879tb\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.538148 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.537953 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00c72a-879tb\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.639166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.639126 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x4wp\" (UniqueName: \"kubernetes.io/projected/221a9e6f-803e-40f8-be5d-4db112c93e4b-kube-api-access-5x4wp\") pod \"maas-api-revoke-keys-e2e-inf-b-00c72a-879tb\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.639363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.639275 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00c72a-879tb\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.639526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.639509 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00c72a-879tb\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.639906 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.639886 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00c72a-879tb\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.641861 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.641832 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00c72a-879tb\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.647214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.647190 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x4wp\" (UniqueName: \"kubernetes.io/projected/221a9e6f-803e-40f8-be5d-4db112c93e4b-kube-api-access-5x4wp\") pod \"maas-api-revoke-keys-e2e-inf-b-00c72a-879tb\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.701298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.701268 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:35.935341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.935309 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn"] Nov 02 22:25:35.936255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:35.936223 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" containerID="cri-o://e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e" gracePeriod=30 Nov 02 22:25:36.198500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:36.198474 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb"] Nov 02 22:25:36.199510 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:36.199484 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221a9e6f_803e_40f8_be5d_4db112c93e4b.slice/crio-70bdcd5c441f17894504c246d040a7a6ae7875fba8665d8256d57103dafbacc5 WatchSource:0}: Error finding container 70bdcd5c441f17894504c246d040a7a6ae7875fba8665d8256d57103dafbacc5: Status 404 returned error can't find the container with id 70bdcd5c441f17894504c246d040a7a6ae7875fba8665d8256d57103dafbacc5 Nov 02 22:25:36.577401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:36.574073 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" event={"ID":"221a9e6f-803e-40f8-be5d-4db112c93e4b","Type":"ContainerStarted","Data":"70bdcd5c441f17894504c246d040a7a6ae7875fba8665d8256d57103dafbacc5"} Nov 02 22:25:37.021911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.021887 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:25:37.095537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.095442 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38542064-c103-49d1-9ce6-f0fd1398ac97-kube-api-access-cwf57\" (UniqueName: \"kubernetes.io/projected/38542064-c103-49d1-9ce6-f0fd1398ac97-kube-api-access-cwf57\") pod \"38542064-c103-49d1-9ce6-f0fd1398ac97\" (UID: \"38542064-c103-49d1-9ce6-f0fd1398ac97\") " Nov 02 22:25:37.095537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.095514 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38542064-c103-49d1-9ce6-f0fd1398ac97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38542064-c103-49d1-9ce6-f0fd1398ac97-plugins-config-volume\") pod \"38542064-c103-49d1-9ce6-f0fd1398ac97\" (UID: \"38542064-c103-49d1-9ce6-f0fd1398ac97\") " Nov 02 22:25:37.096981 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.096953 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38542064-c103-49d1-9ce6-f0fd1398ac97-plugins-config-volume" pod "38542064-c103-49d1-9ce6-f0fd1398ac97" (UID: "38542064-c103-49d1-9ce6-f0fd1398ac97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:37.099722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.099650 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38542064-c103-49d1-9ce6-f0fd1398ac97-kube-api-access-cwf57" pod "38542064-c103-49d1-9ce6-f0fd1398ac97" (UID: "38542064-c103-49d1-9ce6-f0fd1398ac97"). InnerVolumeSpecName "kube-api-access-cwf57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:37.207076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.207002 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwf57\" (UniqueName: \"kubernetes.io/projected/38542064-c103-49d1-9ce6-f0fd1398ac97-kube-api-access-cwf57\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.207076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.207044 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38542064-c103-49d1-9ce6-f0fd1398ac97-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:37.347175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.346750 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf"] Nov 02 22:25:37.353725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.352368 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-df8835-c8qgf"] Nov 02 22:25:37.585589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.585554 4309 generic.go:356] "Generic (PLEG): container finished" podID="38542064-c103-49d1-9ce6-f0fd1398ac97" containerID="0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa" exitCode=0 Nov 02 22:25:37.585780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.585602 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" Nov 02 22:25:37.585780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.585626 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" event={"ID":"38542064-c103-49d1-9ce6-f0fd1398ac97","Type":"ContainerDied","Data":"0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa"} Nov 02 22:25:37.585780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.585730 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8" event={"ID":"38542064-c103-49d1-9ce6-f0fd1398ac97","Type":"ContainerDied","Data":"077bb4e9e7ef48a0da9bd3b2725eb9cde7eada889e5c6be9d0752da947aa1d40"} Nov 02 22:25:37.585780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.585746 4309 scope.go:122] "RemoveContainer" containerID="0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa" Nov 02 22:25:37.587833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.587805 4309 generic.go:356] "Generic (PLEG): container finished" podID="221a9e6f-803e-40f8-be5d-4db112c93e4b" containerID="3f0cbe17ab816caf7ad69bb8b7b2d4023d17d7b55c52ab6375bc135f40b0edc4" exitCode=0 Nov 02 22:25:37.588012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.587994 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" event={"ID":"221a9e6f-803e-40f8-be5d-4db112c93e4b","Type":"ContainerDied","Data":"3f0cbe17ab816caf7ad69bb8b7b2d4023d17d7b55c52ab6375bc135f40b0edc4"} Nov 02 22:25:37.605215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.605145 4309 scope.go:122] "RemoveContainer" containerID="0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa" Nov 02 22:25:37.605426 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:37.605404 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa\": container with ID starting with 0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa not found: ID does not exist" containerID="0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa" Nov 02 22:25:37.605503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.605435 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa"} err="failed to get container status \"0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa\": rpc error: code = NotFound desc = could not find container \"0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa\": container with ID starting with 0a99a3ead44aba4c175cf7bd17a73c85699f938d7857e263b76aee76fdb5bcaa not found: ID does not exist" Nov 02 22:25:37.631631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.631599 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8"] Nov 02 22:25:37.635875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:37.635848 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e49edd-97f8765cf-tw9w8"] Nov 02 22:25:38.193298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.193269 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38542064-c103-49d1-9ce6-f0fd1398ac97" path="/var/lib/kubelet/pods/38542064-c103-49d1-9ce6-f0fd1398ac97/volumes" Nov 02 22:25:38.193910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.193894 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef72440-4080-4344-a5b1-236c6b4b84c0" path="/var/lib/kubelet/pods/6ef72440-4080-4344-a5b1-236c6b4b84c0/volumes" Nov 02 22:25:38.785726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.785665 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft"] Nov 02 22:25:38.785973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.785921 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" containerID="cri-o://d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c" gracePeriod=30 Nov 02 22:25:38.799235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.799172 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:38.847293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.847260 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221a9e6f-803e-40f8-be5d-4db112c93e4b-kube-api-access-5x4wp\" (UniqueName: \"kubernetes.io/projected/221a9e6f-803e-40f8-be5d-4db112c93e4b-kube-api-access-5x4wp\") pod \"221a9e6f-803e-40f8-be5d-4db112c93e4b\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " Nov 02 22:25:38.847514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.847309 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-xks-ca\") pod \"221a9e6f-803e-40f8-be5d-4db112c93e4b\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " Nov 02 22:25:38.847514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.847405 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-openshift-ca\") pod \"221a9e6f-803e-40f8-be5d-4db112c93e4b\" (UID: \"221a9e6f-803e-40f8-be5d-4db112c93e4b\") " Nov 02 22:25:38.848191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.848162 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-openshift-ca" pod "221a9e6f-803e-40f8-be5d-4db112c93e4b" (UID: "221a9e6f-803e-40f8-be5d-4db112c93e4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:38.850312 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.850285 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-xks-ca" pod "221a9e6f-803e-40f8-be5d-4db112c93e4b" (UID: "221a9e6f-803e-40f8-be5d-4db112c93e4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:38.851800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.851776 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221a9e6f-803e-40f8-be5d-4db112c93e4b-kube-api-access-5x4wp" pod "221a9e6f-803e-40f8-be5d-4db112c93e4b" (UID: "221a9e6f-803e-40f8-be5d-4db112c93e4b"). InnerVolumeSpecName "kube-api-access-5x4wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:38.957696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.950937 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x4wp\" (UniqueName: \"kubernetes.io/projected/221a9e6f-803e-40f8-be5d-4db112c93e4b-kube-api-access-5x4wp\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:38.957696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.950983 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:38.957696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:38.950998 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/221a9e6f-803e-40f8-be5d-4db112c93e4b-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:39.278495 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.278409 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5"] Nov 02 22:25:39.279228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.279200 4309 cpu_manager.go:414] "containerMap: removing container" podUID="38542064-c103-49d1-9ce6-f0fd1398ac97" containerName="payload-pre-processing" Nov 02 22:25:39.279228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.279223 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38542064-c103-49d1-9ce6-f0fd1398ac97" containerName="payload-pre-processing" Nov 02 22:25:39.279443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.279268 4309 cpu_manager.go:414] "containerMap: removing container" podUID="221a9e6f-803e-40f8-be5d-4db112c93e4b" containerName="revoke-keys" Nov 02 22:25:39.279443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.279276 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221a9e6f-803e-40f8-be5d-4db112c93e4b" containerName="revoke-keys" Nov 02 22:25:39.279443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.279411 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="38542064-c103-49d1-9ce6-f0fd1398ac97" containerName="payload-pre-processing" Nov 02 22:25:39.279443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.279438 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="221a9e6f-803e-40f8-be5d-4db112c93e4b" containerName="revoke-keys" Nov 02 22:25:39.283283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.283265 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.292486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.292460 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5"] Nov 02 22:25:39.358851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.358819 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.359070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.359045 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.359125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.359087 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6dn\" (UniqueName: \"kubernetes.io/projected/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-kube-api-access-nf6dn\") pod \"maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.387614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.387586 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2"] Nov 02 22:25:39.462747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.460761 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.462747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.460827 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6dn\" (UniqueName: \"kubernetes.io/projected/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-kube-api-access-nf6dn\") pod \"maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.462747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.461029 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.464040 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.464006 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.474557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.472541 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6dn\" (UniqueName: \"kubernetes.io/projected/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-kube-api-access-nf6dn\") pod \"maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.475324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.475273 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.604575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.604549 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:39.607108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.607082 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" event={"ID":"221a9e6f-803e-40f8-be5d-4db112c93e4b","Type":"ContainerDied","Data":"70bdcd5c441f17894504c246d040a7a6ae7875fba8665d8256d57103dafbacc5"} Nov 02 22:25:39.607108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.607128 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70bdcd5c441f17894504c246d040a7a6ae7875fba8665d8256d57103dafbacc5" Nov 02 22:25:39.607108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.607090 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb" Nov 02 22:25:39.934061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:39.934033 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5"] Nov 02 22:25:39.935492 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:39.935451 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014ce160_7bc9_4b3b_9dc1_64a34697c3c7.slice/crio-e07382ec76daa4550e6220addd6f13f1eb72d4e3efda64cf36b7ef41d0a33156 WatchSource:0}: Error finding container e07382ec76daa4550e6220addd6f13f1eb72d4e3efda64cf36b7ef41d0a33156: Status 404 returned error can't find the container with id e07382ec76daa4550e6220addd6f13f1eb72d4e3efda64cf36b7ef41d0a33156 Nov 02 22:25:40.617611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:40.617577 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" event={"ID":"014ce160-7bc9-4b3b-9dc1-64a34697c3c7","Type":"ContainerStarted","Data":"16fd2e028993bb03f171e96fb78dc5a16fddac095010727769dc327d0bc521ff"} Nov 02 22:25:40.618021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:40.617630 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" event={"ID":"014ce160-7bc9-4b3b-9dc1-64a34697c3c7","Type":"ContainerStarted","Data":"e07382ec76daa4550e6220addd6f13f1eb72d4e3efda64cf36b7ef41d0a33156"} Nov 02 22:25:40.633417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:40.633366 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" podStartSLOduration=1.633354079 podStartE2EDuration="1.633354079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:40.631125473 +0000 UTC m=+1668.569238690" watchObservedRunningTime="2026-11-02 22:25:40.633354079 +0000 UTC m=+1668.571467296" Nov 02 22:25:40.907200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:40.907120 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm"] Nov 02 22:25:40.914274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:40.914244 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d28bd1-fq5rm"] Nov 02 22:25:41.206731 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.206657 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:25:41.306062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.306033 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:25:41.315285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.315256 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e97fdbf-c24b-4727-b895-18915dc6b1a0-kube-api-access-nhzq6\" (UniqueName: \"kubernetes.io/projected/6e97fdbf-c24b-4727-b895-18915dc6b1a0-kube-api-access-nhzq6\") pod \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\" (UID: \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\") " Nov 02 22:25:41.315433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.315393 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e97fdbf-c24b-4727-b895-18915dc6b1a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e97fdbf-c24b-4727-b895-18915dc6b1a0-plugins-config-volume\") pod \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\" (UID: \"6e97fdbf-c24b-4727-b895-18915dc6b1a0\") " Nov 02 22:25:41.318179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.318153 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e97fdbf-c24b-4727-b895-18915dc6b1a0-kube-api-access-nhzq6" pod "6e97fdbf-c24b-4727-b895-18915dc6b1a0" (UID: "6e97fdbf-c24b-4727-b895-18915dc6b1a0"). InnerVolumeSpecName "kube-api-access-nhzq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:41.318406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.318391 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e97fdbf-c24b-4727-b895-18915dc6b1a0-plugins-config-volume" pod "6e97fdbf-c24b-4727-b895-18915dc6b1a0" (UID: "6e97fdbf-c24b-4727-b895-18915dc6b1a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:41.417018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.416814 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3a00cf-3743-4d16-932d-99c767e58d83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3a00cf-3743-4d16-932d-99c767e58d83-istiod-ca-cert\") pod \"4e3a00cf-3743-4d16-932d-99c767e58d83\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " Nov 02 22:25:41.417018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.416934 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-socket\") pod \"4e3a00cf-3743-4d16-932d-99c767e58d83\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " Nov 02 22:25:41.417018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.416982 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-credential-socket\") pod \"4e3a00cf-3743-4d16-932d-99c767e58d83\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " Nov 02 22:25:41.417018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.417009 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-data\") pod \"4e3a00cf-3743-4d16-932d-99c767e58d83\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " Nov 02 22:25:41.417271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.417098 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-podinfo\") pod \"4e3a00cf-3743-4d16-932d-99c767e58d83\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " Nov 02 22:25:41.417271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.417145 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-kube-api-access-q26nw\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-kube-api-access-q26nw\") pod \"4e3a00cf-3743-4d16-932d-99c767e58d83\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " Nov 02 22:25:41.417271 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.417211 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-token\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-token\") pod \"4e3a00cf-3743-4d16-932d-99c767e58d83\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " Nov 02 22:25:41.417378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.417299 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-certs\") pod \"4e3a00cf-3743-4d16-932d-99c767e58d83\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " Nov 02 22:25:41.417378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.417316 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-envoy\") pod \"4e3a00cf-3743-4d16-932d-99c767e58d83\" (UID: \"4e3a00cf-3743-4d16-932d-99c767e58d83\") " Nov 02 22:25:41.417835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.417819 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhzq6\" (UniqueName: \"kubernetes.io/projected/6e97fdbf-c24b-4727-b895-18915dc6b1a0-kube-api-access-nhzq6\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.417904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.417837 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e97fdbf-c24b-4727-b895-18915dc6b1a0-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.420588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.420532 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-envoy" pod "4e3a00cf-3743-4d16-932d-99c767e58d83" (UID: "4e3a00cf-3743-4d16-932d-99c767e58d83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:41.420977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.420954 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3a00cf-3743-4d16-932d-99c767e58d83-istiod-ca-cert" pod "4e3a00cf-3743-4d16-932d-99c767e58d83" (UID: "4e3a00cf-3743-4d16-932d-99c767e58d83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:41.421236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.421216 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-socket" pod "4e3a00cf-3743-4d16-932d-99c767e58d83" (UID: "4e3a00cf-3743-4d16-932d-99c767e58d83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:41.421407 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.421382 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-credential-socket" pod "4e3a00cf-3743-4d16-932d-99c767e58d83" (UID: "4e3a00cf-3743-4d16-932d-99c767e58d83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:41.421566 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.421549 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-data" pod "4e3a00cf-3743-4d16-932d-99c767e58d83" (UID: "4e3a00cf-3743-4d16-932d-99c767e58d83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:41.423169 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.423142 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-certs" pod "4e3a00cf-3743-4d16-932d-99c767e58d83" (UID: "4e3a00cf-3743-4d16-932d-99c767e58d83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:41.423766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.423740 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-podinfo" pod "4e3a00cf-3743-4d16-932d-99c767e58d83" (UID: "4e3a00cf-3743-4d16-932d-99c767e58d83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:41.424764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.424741 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-token" pod "4e3a00cf-3743-4d16-932d-99c767e58d83" (UID: "4e3a00cf-3743-4d16-932d-99c767e58d83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:41.425303 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.425277 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-kube-api-access-q26nw" pod "4e3a00cf-3743-4d16-932d-99c767e58d83" (UID: "4e3a00cf-3743-4d16-932d-99c767e58d83"). InnerVolumeSpecName "kube-api-access-q26nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:41.519666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.519623 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.519666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.519661 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.519666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.519690 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3a00cf-3743-4d16-932d-99c767e58d83-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.519666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.519703 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.520066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.519721 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.520066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.519734 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.520066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.519747 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.520066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.519762 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q26nw\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-kube-api-access-q26nw\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.520066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.519776 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3a00cf-3743-4d16-932d-99c767e58d83-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:41.601076 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:41.601035 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:25:41.628665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.628623 4309 generic.go:356] "Generic (PLEG): container finished" podID="014ce160-7bc9-4b3b-9dc1-64a34697c3c7" containerID="16fd2e028993bb03f171e96fb78dc5a16fddac095010727769dc327d0bc521ff" exitCode=0 Nov 02 22:25:41.629138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.628724 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" event={"ID":"014ce160-7bc9-4b3b-9dc1-64a34697c3c7","Type":"ContainerDied","Data":"16fd2e028993bb03f171e96fb78dc5a16fddac095010727769dc327d0bc521ff"} Nov 02 22:25:41.630354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.630331 4309 generic.go:356] "Generic (PLEG): container finished" podID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerID="e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e" exitCode=0 Nov 02 22:25:41.630449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.630377 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" Nov 02 22:25:41.630449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.630410 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" event={"ID":"4e3a00cf-3743-4d16-932d-99c767e58d83","Type":"ContainerDied","Data":"e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e"} Nov 02 22:25:41.630541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.630460 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn" event={"ID":"4e3a00cf-3743-4d16-932d-99c767e58d83","Type":"ContainerDied","Data":"ec1ebe848c21708db5afcf3bd8b973bfc74a58d7aa66cdf39dfd03a623dfd78f"} Nov 02 22:25:41.630541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.630488 4309 scope.go:122] "RemoveContainer" containerID="e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e" Nov 02 22:25:41.632943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.632347 4309 generic.go:356] "Generic (PLEG): container finished" podID="6e97fdbf-c24b-4727-b895-18915dc6b1a0" containerID="a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a" exitCode=0 Nov 02 22:25:41.633426 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.633402 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" Nov 02 22:25:41.633513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.633450 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" event={"ID":"6e97fdbf-c24b-4727-b895-18915dc6b1a0","Type":"ContainerDied","Data":"a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a"} Nov 02 22:25:41.633513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.633496 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd" event={"ID":"6e97fdbf-c24b-4727-b895-18915dc6b1a0","Type":"ContainerDied","Data":"da0eb3b6457d2d54304d7788c03f61b895284cad3b07cb1ab0b1ef710ccb25af"} Nov 02 22:25:41.648391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.648369 4309 scope.go:122] "RemoveContainer" containerID="e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e" Nov 02 22:25:41.648776 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:41.648749 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e\": container with ID starting with e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e not found: ID does not exist" containerID="e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e" Nov 02 22:25:41.648868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.648785 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e"} err="failed to get container status \"e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e\": rpc error: code = NotFound desc = could not find container \"e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e\": container with ID starting with e79861a9b5be865dd8dbceb38fb20777bbb21c5995aede51d45d0bd270ad0e7e not found: ID does not exist" Nov 02 22:25:41.648868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.648810 4309 scope.go:122] "RemoveContainer" containerID="a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a" Nov 02 22:25:41.662635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.662615 4309 scope.go:122] "RemoveContainer" containerID="a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a" Nov 02 22:25:41.663538 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:41.663017 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a\": container with ID starting with a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a not found: ID does not exist" containerID="a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a" Nov 02 22:25:41.663538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.663061 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a"} err="failed to get container status \"a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a\": rpc error: code = NotFound desc = could not find container \"a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a\": container with ID starting with a0d5bcac160ee87b840dd73dd7c9e8e92c541a5f482fa7e9669181f5262e844a not found: ID does not exist" Nov 02 22:25:41.667586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.667526 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn"] Nov 02 22:25:41.672643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.672622 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-8vgmn"] Nov 02 22:25:41.685890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.685863 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd"] Nov 02 22:25:41.690130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:41.690107 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e30968-64fcd7c995-b29wd"] Nov 02 22:25:42.202475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.202427 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c26468-887a-4ac2-ba5b-2e9cc91ffe2e" path="/var/lib/kubelet/pods/30c26468-887a-4ac2-ba5b-2e9cc91ffe2e/volumes" Nov 02 22:25:42.203052 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.203030 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" path="/var/lib/kubelet/pods/4e3a00cf-3743-4d16-932d-99c767e58d83/volumes" Nov 02 22:25:42.203524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.203504 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e97fdbf-c24b-4727-b895-18915dc6b1a0" path="/var/lib/kubelet/pods/6e97fdbf-c24b-4727-b895-18915dc6b1a0/volumes" Nov 02 22:25:42.820103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.820067 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:42.949011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.948924 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-openshift-ca\") pod \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " Nov 02 22:25:42.949184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.949023 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-xks-ca\") pod \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " Nov 02 22:25:42.949241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.949229 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-kube-api-access-nf6dn\" (UniqueName: \"kubernetes.io/projected/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-kube-api-access-nf6dn\") pod \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\" (UID: \"014ce160-7bc9-4b3b-9dc1-64a34697c3c7\") " Nov 02 22:25:42.949362 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.949340 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-openshift-ca" pod "014ce160-7bc9-4b3b-9dc1-64a34697c3c7" (UID: "014ce160-7bc9-4b3b-9dc1-64a34697c3c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:42.949891 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.949865 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:42.951061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.951032 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-xks-ca" pod "014ce160-7bc9-4b3b-9dc1-64a34697c3c7" (UID: "014ce160-7bc9-4b3b-9dc1-64a34697c3c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:42.951199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:42.951182 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-kube-api-access-nf6dn" pod "014ce160-7bc9-4b3b-9dc1-64a34697c3c7" (UID: "014ce160-7bc9-4b3b-9dc1-64a34697c3c7"). InnerVolumeSpecName "kube-api-access-nf6dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:43.051400 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:43.051365 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf6dn\" (UniqueName: \"kubernetes.io/projected/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-kube-api-access-nf6dn\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:43.051400 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:43.051395 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/014ce160-7bc9-4b3b-9dc1-64a34697c3c7-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:43.742772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:43.734157 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" event={"ID":"014ce160-7bc9-4b3b-9dc1-64a34697c3c7","Type":"ContainerDied","Data":"e07382ec76daa4550e6220addd6f13f1eb72d4e3efda64cf36b7ef41d0a33156"} Nov 02 22:25:43.742772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:43.734224 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e07382ec76daa4550e6220addd6f13f1eb72d4e3efda64cf36b7ef41d0a33156" Nov 02 22:25:43.742772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:43.734289 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5" Nov 02 22:25:44.075562 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.075537 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:25:44.090706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.090640 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-envoy\") pod \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " Nov 02 22:25:44.090874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.090713 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istiod-ca-cert\") pod \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " Nov 02 22:25:44.090874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.090768 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-socket\") pod \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " Nov 02 22:25:44.090874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.090803 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-certs\") pod \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " Nov 02 22:25:44.090874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.090827 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-credential-socket\") pod \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " Nov 02 22:25:44.091112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.090874 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-data\") pod \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " Nov 02 22:25:44.091112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.090902 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-token\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-token\") pod \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " Nov 02 22:25:44.091112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091039 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-podinfo\") pod \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " Nov 02 22:25:44.091112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091047 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-socket" pod "b4f752b0-65ba-4b65-b25d-3506c2cf0128" (UID: "b4f752b0-65ba-4b65-b25d-3506c2cf0128"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:44.091112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091075 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-certs" pod "b4f752b0-65ba-4b65-b25d-3506c2cf0128" (UID: "b4f752b0-65ba-4b65-b25d-3506c2cf0128"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:44.091389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091141 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istiod-ca-cert" pod "b4f752b0-65ba-4b65-b25d-3506c2cf0128" (UID: "b4f752b0-65ba-4b65-b25d-3506c2cf0128"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:44.091389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091207 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-kube-api-access-nwtv5\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-kube-api-access-nwtv5\") pod \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\" (UID: \"b4f752b0-65ba-4b65-b25d-3506c2cf0128\") " Nov 02 22:25:44.093254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091494 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-credential-socket" pod "b4f752b0-65ba-4b65-b25d-3506c2cf0128" (UID: "b4f752b0-65ba-4b65-b25d-3506c2cf0128"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:44.093254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091753 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-data" pod "b4f752b0-65ba-4b65-b25d-3506c2cf0128" (UID: "b4f752b0-65ba-4b65-b25d-3506c2cf0128"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:44.093254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091855 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.093254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091877 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.093254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091892 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.093254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091907 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.093254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.091920 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.094872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.094724 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-envoy" pod "b4f752b0-65ba-4b65-b25d-3506c2cf0128" (UID: "b4f752b0-65ba-4b65-b25d-3506c2cf0128"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:44.094872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.094742 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-kube-api-access-nwtv5" pod "b4f752b0-65ba-4b65-b25d-3506c2cf0128" (UID: "b4f752b0-65ba-4b65-b25d-3506c2cf0128"). InnerVolumeSpecName "kube-api-access-nwtv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:44.095065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.095043 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-token" pod "b4f752b0-65ba-4b65-b25d-3506c2cf0128" (UID: "b4f752b0-65ba-4b65-b25d-3506c2cf0128"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:44.102398 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.101871 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-podinfo" pod "b4f752b0-65ba-4b65-b25d-3506c2cf0128" (UID: "b4f752b0-65ba-4b65-b25d-3506c2cf0128"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:44.193453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.193075 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtv5\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-kube-api-access-nwtv5\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.193453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.193107 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.193453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.193122 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.193453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.193136 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f752b0-65ba-4b65-b25d-3506c2cf0128-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.388726 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.388639 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" podUID="357afe48-05c2-4529-b51f-196c8aacc233" containerName="payload-processing" containerID="cri-o://48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac" gracePeriod=25 Nov 02 22:25:44.707900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.707873 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:25:44.751177 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.751146 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549"] Nov 02 22:25:44.751594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.751561 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" containerID="cri-o://521b28391ffab3c26b343332c1879d98e78786f7b001e0cf312d16a99077d4d7" gracePeriod=30 Nov 02 22:25:44.760772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.759839 4309 generic.go:356] "Generic (PLEG): container finished" podID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerID="d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c" exitCode=0 Nov 02 22:25:44.760772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.759958 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" Nov 02 22:25:44.760772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.760188 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" event={"ID":"b4f752b0-65ba-4b65-b25d-3506c2cf0128","Type":"ContainerDied","Data":"d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c"} Nov 02 22:25:44.760772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.760231 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft" event={"ID":"b4f752b0-65ba-4b65-b25d-3506c2cf0128","Type":"ContainerDied","Data":"6e9cac88582304dc6fab55c313fb3a9838039d10f37369ba51449882da95c0c9"} Nov 02 22:25:44.760772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.760257 4309 scope.go:122] "RemoveContainer" containerID="d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c" Nov 02 22:25:44.763899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.762811 4309 generic.go:356] "Generic (PLEG): container finished" podID="357afe48-05c2-4529-b51f-196c8aacc233" containerID="48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac" exitCode=0 Nov 02 22:25:44.763899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.762912 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" event={"ID":"357afe48-05c2-4529-b51f-196c8aacc233","Type":"ContainerDied","Data":"48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac"} Nov 02 22:25:44.763899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.762950 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" event={"ID":"357afe48-05c2-4529-b51f-196c8aacc233","Type":"ContainerDied","Data":"9720420658f94929c2f0acdeee8cc0017b534482a0341c9864894959375e0365"} Nov 02 22:25:44.763899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.763018 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2" Nov 02 22:25:44.803060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.803020 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357afe48-05c2-4529-b51f-196c8aacc233-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357afe48-05c2-4529-b51f-196c8aacc233-plugins-config-volume\") pod \"357afe48-05c2-4529-b51f-196c8aacc233\" (UID: \"357afe48-05c2-4529-b51f-196c8aacc233\") " Nov 02 22:25:44.803433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.803402 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357afe48-05c2-4529-b51f-196c8aacc233-kube-api-access-jmmvv\" (UniqueName: \"kubernetes.io/projected/357afe48-05c2-4529-b51f-196c8aacc233-kube-api-access-jmmvv\") pod \"357afe48-05c2-4529-b51f-196c8aacc233\" (UID: \"357afe48-05c2-4529-b51f-196c8aacc233\") " Nov 02 22:25:44.814003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.813319 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357afe48-05c2-4529-b51f-196c8aacc233-plugins-config-volume" pod "357afe48-05c2-4529-b51f-196c8aacc233" (UID: "357afe48-05c2-4529-b51f-196c8aacc233"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:44.814483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.814463 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357afe48-05c2-4529-b51f-196c8aacc233-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.821875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.821849 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357afe48-05c2-4529-b51f-196c8aacc233-kube-api-access-jmmvv" pod "357afe48-05c2-4529-b51f-196c8aacc233" (UID: "357afe48-05c2-4529-b51f-196c8aacc233"). InnerVolumeSpecName "kube-api-access-jmmvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:44.823013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.822987 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft"] Nov 02 22:25:44.828902 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.828880 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a54217-openshift-default-554d688f75-qlbft"] Nov 02 22:25:44.834246 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.834218 4309 scope.go:122] "RemoveContainer" containerID="372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1" Nov 02 22:25:44.913219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.907309 4309 scope.go:122] "RemoveContainer" containerID="d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c" Nov 02 22:25:44.913219 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:44.909155 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c\": container with ID starting with d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c not found: ID does not exist" containerID="d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c" Nov 02 22:25:44.913219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.909194 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c"} err="failed to get container status \"d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c\": rpc error: code = NotFound desc = could not find container \"d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c\": container with ID starting with d9faddff1a102a88acd01e4adbd21a457c973d6be6443a3608f34d9ce8cb7f6c not found: ID does not exist" Nov 02 22:25:44.913219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.909222 4309 scope.go:122] "RemoveContainer" containerID="372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1" Nov 02 22:25:44.913219 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:44.909496 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1\": container with ID starting with 372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1 not found: ID does not exist" containerID="372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1" Nov 02 22:25:44.913219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.909520 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1"} err="failed to get container status \"372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1\": rpc error: code = NotFound desc = could not find container \"372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1\": container with ID starting with 372d48fce6fb46b9d6dfc003d93ba7dac0eaeee6fc31d2aca648c50abb4462a1 not found: ID does not exist" Nov 02 22:25:44.913219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.909549 4309 scope.go:122] "RemoveContainer" containerID="48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac" Nov 02 22:25:44.916278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.916236 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmvv\" (UniqueName: \"kubernetes.io/projected/357afe48-05c2-4529-b51f-196c8aacc233-kube-api-access-jmmvv\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:44.944012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.943639 4309 scope.go:122] "RemoveContainer" containerID="48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac" Nov 02 22:25:44.944354 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:44.944259 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac\": container with ID starting with 48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac not found: ID does not exist" containerID="48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac" Nov 02 22:25:44.944354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:44.944296 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac"} err="failed to get container status \"48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac\": rpc error: code = NotFound desc = could not find container \"48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac\": container with ID starting with 48ca340712b85a0d3a8619eaa73065d55239268f5609b9026834569f3f1f1cac not found: ID does not exist" Nov 02 22:25:45.026357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.025230 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" podUID="ffc1bd6d-fcec-4658-9968-4bd9bd401d15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:25:45.089357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.089322 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2"] Nov 02 22:25:45.093917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.093889 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d28bd1-75d6969d95-76gr2"] Nov 02 22:25:45.236712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.236612 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6"] Nov 02 22:25:45.237400 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237378 4309 cpu_manager.go:414] "containerMap: removing container" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" Nov 02 22:25:45.237527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237400 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" Nov 02 22:25:45.237527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237424 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" Nov 02 22:25:45.237527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237431 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" Nov 02 22:25:45.237527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237442 4309 cpu_manager.go:414] "containerMap: removing container" podUID="357afe48-05c2-4529-b51f-196c8aacc233" containerName="payload-processing" Nov 02 22:25:45.237527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237449 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357afe48-05c2-4529-b51f-196c8aacc233" containerName="payload-processing" Nov 02 22:25:45.237527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237466 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" Nov 02 22:25:45.237527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237473 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" Nov 02 22:25:45.237527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237497 4309 cpu_manager.go:414] "containerMap: removing container" podUID="014ce160-7bc9-4b3b-9dc1-64a34697c3c7" containerName="revoke-keys" Nov 02 22:25:45.237527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237503 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014ce160-7bc9-4b3b-9dc1-64a34697c3c7" containerName="revoke-keys" Nov 02 22:25:45.238011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237566 4309 cpu_manager.go:414] "containerMap: removing container" podUID="6e97fdbf-c24b-4727-b895-18915dc6b1a0" containerName="payload-pre-processing" Nov 02 22:25:45.238011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237575 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e97fdbf-c24b-4727-b895-18915dc6b1a0" containerName="payload-pre-processing" Nov 02 22:25:45.238011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237764 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3a00cf-3743-4d16-932d-99c767e58d83" containerName="istio-proxy" Nov 02 22:25:45.238011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237782 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" Nov 02 22:25:45.238011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237794 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="357afe48-05c2-4529-b51f-196c8aacc233" containerName="payload-processing" Nov 02 22:25:45.238011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237804 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" containerName="istio-proxy" Nov 02 22:25:45.238011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237815 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="014ce160-7bc9-4b3b-9dc1-64a34697c3c7" containerName="revoke-keys" Nov 02 22:25:45.238011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.237833 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e97fdbf-c24b-4727-b895-18915dc6b1a0" containerName="payload-pre-processing" Nov 02 22:25:45.241648 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.241628 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.245212 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.245192 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:25:45.245792 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.245774 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:25:45.247986 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.247950 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6"] Nov 02 22:25:45.323036 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.323007 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:45.323388 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.323368 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:25:45.329190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.329167 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.329386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.329369 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.329455 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.329406 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8ft\" (UniqueName: \"kubernetes.io/projected/7b867ca1-8d8e-45de-9124-0186a7d23373-kube-api-access-vx8ft\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.430639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.430602 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.430828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.430763 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.430828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.430793 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8ft\" (UniqueName: \"kubernetes.io/projected/7b867ca1-8d8e-45de-9124-0186a7d23373-kube-api-access-vx8ft\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.431474 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.431442 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.433240 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.433217 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.441449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.441426 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8ft\" (UniqueName: \"kubernetes.io/projected/7b867ca1-8d8e-45de-9124-0186a7d23373-kube-api-access-vx8ft\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.561751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.561719 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:45.954808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:45.954766 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6"] Nov 02 22:25:46.191088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:46.191003 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357afe48-05c2-4529-b51f-196c8aacc233" path="/var/lib/kubelet/pods/357afe48-05c2-4529-b51f-196c8aacc233/volumes" Nov 02 22:25:46.191582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:46.191563 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f752b0-65ba-4b65-b25d-3506c2cf0128" path="/var/lib/kubelet/pods/b4f752b0-65ba-4b65-b25d-3506c2cf0128/volumes" Nov 02 22:25:46.869043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:46.869005 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" event={"ID":"7b867ca1-8d8e-45de-9124-0186a7d23373","Type":"ContainerStarted","Data":"feb8cee33d6448211d0713076e53c379c9681622e24d4eefcc7cd65fbdb9e126"} Nov 02 22:25:46.869208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:46.869067 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" event={"ID":"7b867ca1-8d8e-45de-9124-0186a7d23373","Type":"ContainerStarted","Data":"a234a50ca77632ba8823f6571b53977214c622acf6f9ada03ca304264998c6a4"} Nov 02 22:25:47.880484 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:47.880449 4309 generic.go:356] "Generic (PLEG): container finished" podID="7b867ca1-8d8e-45de-9124-0186a7d23373" containerID="feb8cee33d6448211d0713076e53c379c9681622e24d4eefcc7cd65fbdb9e126" exitCode=0 Nov 02 22:25:47.880877 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:47.880536 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" event={"ID":"7b867ca1-8d8e-45de-9124-0186a7d23373","Type":"ContainerDied","Data":"feb8cee33d6448211d0713076e53c379c9681622e24d4eefcc7cd65fbdb9e126"} Nov 02 22:25:48.506993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.506956 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:25:48.627152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.627113 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww"] Nov 02 22:25:48.627987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.627406 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" containerID="cri-o://68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e" gracePeriod=30 Nov 02 22:25:48.640920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.640890 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-plugins-config-volume\") pod \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\" (UID: \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\") " Nov 02 22:25:48.641055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.640990 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-kube-api-access-4mq9g\" (UniqueName: \"kubernetes.io/projected/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-kube-api-access-4mq9g\") pod \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\" (UID: \"ffc1bd6d-fcec-4658-9968-4bd9bd401d15\") " Nov 02 22:25:48.641508 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.641265 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-plugins-config-volume" pod "ffc1bd6d-fcec-4658-9968-4bd9bd401d15" (UID: "ffc1bd6d-fcec-4658-9968-4bd9bd401d15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:48.641906 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.641875 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:48.644888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.644855 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-kube-api-access-4mq9g" pod "ffc1bd6d-fcec-4658-9968-4bd9bd401d15" (UID: "ffc1bd6d-fcec-4658-9968-4bd9bd401d15"). InnerVolumeSpecName "kube-api-access-4mq9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:48.767751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.765068 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mq9g\" (UniqueName: \"kubernetes.io/projected/ffc1bd6d-fcec-4658-9968-4bd9bd401d15-kube-api-access-4mq9g\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:48.911143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.911105 4309 generic.go:356] "Generic (PLEG): container finished" podID="ffc1bd6d-fcec-4658-9968-4bd9bd401d15" containerID="7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d" exitCode=0 Nov 02 22:25:48.912144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.911647 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" event={"ID":"ffc1bd6d-fcec-4658-9968-4bd9bd401d15","Type":"ContainerDied","Data":"7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d"} Nov 02 22:25:48.912327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.911743 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" Nov 02 22:25:48.914064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.913929 4309 scope.go:122] "RemoveContainer" containerID="7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d" Nov 02 22:25:48.914709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.912318 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w" event={"ID":"ffc1bd6d-fcec-4658-9968-4bd9bd401d15","Type":"ContainerDied","Data":"dcfd7e92fad19ecc7b5b9906fba87d20d3225efc117d4106232512b1f25181ec"} Nov 02 22:25:48.932620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.932591 4309 scope.go:122] "RemoveContainer" containerID="7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d" Nov 02 22:25:48.933916 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:48.933883 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d\": container with ID starting with 7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d not found: ID does not exist" containerID="7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d" Nov 02 22:25:48.934022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.933921 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d"} err="failed to get container status \"7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d\": rpc error: code = NotFound desc = could not find container \"7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d\": container with ID starting with 7b3a80d27bbbf259274fd1695db8b67fc34ed1bcba96d5aa187a1ae16b72220d not found: ID does not exist" Nov 02 22:25:48.945698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.945654 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w"] Nov 02 22:25:48.949656 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:48.949634 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a54217-695f484799-48p4w"] Nov 02 22:25:49.094565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.094526 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p"] Nov 02 22:25:49.095636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.095606 4309 cpu_manager.go:414] "containerMap: removing container" podUID="ffc1bd6d-fcec-4658-9968-4bd9bd401d15" containerName="payload-pre-processing" Nov 02 22:25:49.095636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.095636 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc1bd6d-fcec-4658-9968-4bd9bd401d15" containerName="payload-pre-processing" Nov 02 22:25:49.095895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.095876 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc1bd6d-fcec-4658-9968-4bd9bd401d15" containerName="payload-pre-processing" Nov 02 22:25:49.104486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.104436 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.107657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.107629 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p"] Nov 02 22:25:49.177119 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.177089 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:49.187155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.187123 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.187325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.187267 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.187325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.187292 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxnnz\" (UniqueName: \"kubernetes.io/projected/525c64d4-fb38-4bbd-bec8-42cdb184d7af-kube-api-access-rxnnz\") pod \"maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.288100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.288057 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-xks-ca\") pod \"7b867ca1-8d8e-45de-9124-0186a7d23373\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " Nov 02 22:25:49.288415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.288396 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-openshift-ca\") pod \"7b867ca1-8d8e-45de-9124-0186a7d23373\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " Nov 02 22:25:49.288481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.288454 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b867ca1-8d8e-45de-9124-0186a7d23373-kube-api-access-vx8ft\" (UniqueName: \"kubernetes.io/projected/7b867ca1-8d8e-45de-9124-0186a7d23373-kube-api-access-vx8ft\") pod \"7b867ca1-8d8e-45de-9124-0186a7d23373\" (UID: \"7b867ca1-8d8e-45de-9124-0186a7d23373\") " Nov 02 22:25:49.288764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.288740 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.288864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.288782 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxnnz\" (UniqueName: \"kubernetes.io/projected/525c64d4-fb38-4bbd-bec8-42cdb184d7af-kube-api-access-rxnnz\") pod \"maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.288864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.288829 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-openshift-ca" pod "7b867ca1-8d8e-45de-9124-0186a7d23373" (UID: "7b867ca1-8d8e-45de-9124-0186a7d23373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:49.290235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.289145 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.290235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.290070 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.291279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.291257 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:49.292279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.292255 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-xks-ca" pod "7b867ca1-8d8e-45de-9124-0186a7d23373" (UID: "7b867ca1-8d8e-45de-9124-0186a7d23373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:49.292561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.292541 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b867ca1-8d8e-45de-9124-0186a7d23373-kube-api-access-vx8ft" pod "7b867ca1-8d8e-45de-9124-0186a7d23373" (UID: "7b867ca1-8d8e-45de-9124-0186a7d23373"). InnerVolumeSpecName "kube-api-access-vx8ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:49.293201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.293179 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.298824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.298796 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxnnz\" (UniqueName: \"kubernetes.io/projected/525c64d4-fb38-4bbd-bec8-42cdb184d7af-kube-api-access-rxnnz\") pod \"maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.389493 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.389404 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj"] Nov 02 22:25:49.392853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.392825 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx8ft\" (UniqueName: \"kubernetes.io/projected/7b867ca1-8d8e-45de-9124-0186a7d23373-kube-api-access-vx8ft\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:49.392993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.392857 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b867ca1-8d8e-45de-9124-0186a7d23373-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:49.437468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.437428 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:49.762324 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:49.762289 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525c64d4_fb38_4bbd_bec8_42cdb184d7af.slice/crio-6cc5907b912aa38e686668f87b35561ffaf1c6d3aaa211fc4cfa2d540c422c1b WatchSource:0}: Error finding container 6cc5907b912aa38e686668f87b35561ffaf1c6d3aaa211fc4cfa2d540c422c1b: Status 404 returned error can't find the container with id 6cc5907b912aa38e686668f87b35561ffaf1c6d3aaa211fc4cfa2d540c422c1b Nov 02 22:25:49.762570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.762538 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p"] Nov 02 22:25:49.975582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.964750 4309 generic.go:356] "Generic (PLEG): container finished" podID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerID="521b28391ffab3c26b343332c1879d98e78786f7b001e0cf312d16a99077d4d7" exitCode=0 Nov 02 22:25:49.975582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.964928 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" event={"ID":"a5748639-a17c-4ea1-9400-ac7fd3519cc7","Type":"ContainerDied","Data":"521b28391ffab3c26b343332c1879d98e78786f7b001e0cf312d16a99077d4d7"} Nov 02 22:25:49.975582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.965016 4309 scope.go:122] "RemoveContainer" containerID="dcfc309ca2fa804520d026dd426d75cd01e10f5e03a76b6d8f0e4b88008c6050" Nov 02 22:25:49.975582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.970018 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" event={"ID":"525c64d4-fb38-4bbd-bec8-42cdb184d7af","Type":"ContainerStarted","Data":"6cc5907b912aa38e686668f87b35561ffaf1c6d3aaa211fc4cfa2d540c422c1b"} Nov 02 22:25:49.975582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.971981 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" event={"ID":"7b867ca1-8d8e-45de-9124-0186a7d23373","Type":"ContainerDied","Data":"a234a50ca77632ba8823f6571b53977214c622acf6f9ada03ca304264998c6a4"} Nov 02 22:25:49.975582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.972030 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a234a50ca77632ba8823f6571b53977214c622acf6f9ada03ca304264998c6a4" Nov 02 22:25:49.975582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:49.972093 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6" Nov 02 22:25:50.236717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.236668 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc1bd6d-fcec-4658-9968-4bd9bd401d15" path="/var/lib/kubelet/pods/ffc1bd6d-fcec-4658-9968-4bd9bd401d15/volumes" Nov 02 22:25:50.242814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.242733 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb"] Nov 02 22:25:50.245046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.245012 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00c72a-879tb"] Nov 02 22:25:50.340752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.332967 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:25:50.352829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.352797 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-podinfo\") pod \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " Nov 02 22:25:50.353173 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.353153 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-kube-api-access-jwmqh\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-kube-api-access-jwmqh\") pod \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " Nov 02 22:25:50.353329 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.353311 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-envoy\") pod \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " Nov 02 22:25:50.381507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.374832 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-envoy" pod "a5748639-a17c-4ea1-9400-ac7fd3519cc7" (UID: "a5748639-a17c-4ea1-9400-ac7fd3519cc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:50.395011 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.394956 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-podinfo" pod "a5748639-a17c-4ea1-9400-ac7fd3519cc7" (UID: "a5748639-a17c-4ea1-9400-ac7fd3519cc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:50.397559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.397523 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-kube-api-access-jwmqh" pod "a5748639-a17c-4ea1-9400-ac7fd3519cc7" (UID: "a5748639-a17c-4ea1-9400-ac7fd3519cc7"). InnerVolumeSpecName "kube-api-access-jwmqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:50.455219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455187 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-credential-socket\") pod \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " Nov 02 22:25:50.455403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455320 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-socket\") pod \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " Nov 02 22:25:50.455403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455388 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-data\") pod \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " Nov 02 22:25:50.455490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455441 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istiod-ca-cert\") pod \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " Nov 02 22:25:50.455490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455452 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-credential-socket" pod "a5748639-a17c-4ea1-9400-ac7fd3519cc7" (UID: "a5748639-a17c-4ea1-9400-ac7fd3519cc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:50.455490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455478 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-token\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-token\") pod \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " Nov 02 22:25:50.455586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455556 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-socket" pod "a5748639-a17c-4ea1-9400-ac7fd3519cc7" (UID: "a5748639-a17c-4ea1-9400-ac7fd3519cc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:50.455649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455621 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-certs\") pod \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\" (UID: \"a5748639-a17c-4ea1-9400-ac7fd3519cc7\") " Nov 02 22:25:50.455722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455698 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-data" pod "a5748639-a17c-4ea1-9400-ac7fd3519cc7" (UID: "a5748639-a17c-4ea1-9400-ac7fd3519cc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:50.455826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455795 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istiod-ca-cert" pod "a5748639-a17c-4ea1-9400-ac7fd3519cc7" (UID: "a5748639-a17c-4ea1-9400-ac7fd3519cc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:50.455992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.455972 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-certs" pod "a5748639-a17c-4ea1-9400-ac7fd3519cc7" (UID: "a5748639-a17c-4ea1-9400-ac7fd3519cc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:50.456244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.456228 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:50.456314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.456250 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:50.456314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.456264 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:50.456314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.456281 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:50.456314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.456297 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwmqh\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-kube-api-access-jwmqh\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:50.456314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.456312 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:50.456491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.456325 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:50.456491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.456339 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:50.457605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.457587 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-token" pod "a5748639-a17c-4ea1-9400-ac7fd3519cc7" (UID: "a5748639-a17c-4ea1-9400-ac7fd3519cc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:50.557933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.557897 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5748639-a17c-4ea1-9400-ac7fd3519cc7-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:50.971272 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:50.969525 4309 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525c64d4_fb38_4bbd_bec8_42cdb184d7af.slice/crio-eefacf8613cb8188071cd7fc04713c4baa956f10909df506717bc0b14cb9257c.scope\": RecentStats: unable to find data in memory cache]" Nov 02 22:25:50.983526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.983503 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" Nov 02 22:25:50.984103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.983521 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549" event={"ID":"a5748639-a17c-4ea1-9400-ac7fd3519cc7","Type":"ContainerDied","Data":"76792d9ca4104709c4f2e78c6c7645ba2d99b06867675c6b3cd61fab9efc5540"} Nov 02 22:25:50.984235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.984174 4309 scope.go:122] "RemoveContainer" containerID="521b28391ffab3c26b343332c1879d98e78786f7b001e0cf312d16a99077d4d7" Nov 02 22:25:50.989934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.989896 4309 generic.go:356] "Generic (PLEG): container finished" podID="525c64d4-fb38-4bbd-bec8-42cdb184d7af" containerID="eefacf8613cb8188071cd7fc04713c4baa956f10909df506717bc0b14cb9257c" exitCode=0 Nov 02 22:25:50.989934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:50.989991 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" event={"ID":"525c64d4-fb38-4bbd-bec8-42cdb184d7af","Type":"ContainerDied","Data":"eefacf8613cb8188071cd7fc04713c4baa956f10909df506717bc0b14cb9257c"} Nov 02 22:25:51.081864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:51.081817 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549"] Nov 02 22:25:51.098459 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:51.098425 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-df8835-openshift-default-57d5f9546c-pt549"] Nov 02 22:25:52.175623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.175594 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:52.190766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.190736 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221a9e6f-803e-40f8-be5d-4db112c93e4b" path="/var/lib/kubelet/pods/221a9e6f-803e-40f8-be5d-4db112c93e4b/volumes" Nov 02 22:25:52.191559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.191339 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" path="/var/lib/kubelet/pods/a5748639-a17c-4ea1-9400-ac7fd3519cc7/volumes" Nov 02 22:25:52.237147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.237110 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-openshift-ca\") pod \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " Nov 02 22:25:52.239029 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.237283 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-xks-ca\") pod \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " Nov 02 22:25:52.239029 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.237497 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525c64d4-fb38-4bbd-bec8-42cdb184d7af-kube-api-access-rxnnz\" (UniqueName: \"kubernetes.io/projected/525c64d4-fb38-4bbd-bec8-42cdb184d7af-kube-api-access-rxnnz\") pod \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\" (UID: \"525c64d4-fb38-4bbd-bec8-42cdb184d7af\") " Nov 02 22:25:52.239029 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.237845 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-openshift-ca" pod "525c64d4-fb38-4bbd-bec8-42cdb184d7af" (UID: "525c64d4-fb38-4bbd-bec8-42cdb184d7af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:52.239029 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.238899 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:52.243925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.243899 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525c64d4-fb38-4bbd-bec8-42cdb184d7af-kube-api-access-rxnnz" pod "525c64d4-fb38-4bbd-bec8-42cdb184d7af" (UID: "525c64d4-fb38-4bbd-bec8-42cdb184d7af"). InnerVolumeSpecName "kube-api-access-rxnnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:52.244049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.243966 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-xks-ca" pod "525c64d4-fb38-4bbd-bec8-42cdb184d7af" (UID: "525c64d4-fb38-4bbd-bec8-42cdb184d7af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:25:52.340370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.340342 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxnnz\" (UniqueName: \"kubernetes.io/projected/525c64d4-fb38-4bbd-bec8-42cdb184d7af-kube-api-access-rxnnz\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:52.340370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:52.340369 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525c64d4-fb38-4bbd-bec8-42cdb184d7af-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:53.042180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.042144 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" event={"ID":"525c64d4-fb38-4bbd-bec8-42cdb184d7af","Type":"ContainerDied","Data":"6cc5907b912aa38e686668f87b35561ffaf1c6d3aaa211fc4cfa2d540c422c1b"} Nov 02 22:25:53.042359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.042197 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cc5907b912aa38e686668f87b35561ffaf1c6d3aaa211fc4cfa2d540c422c1b" Nov 02 22:25:53.042359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.042246 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p" Nov 02 22:25:53.330699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.321695 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5"] Nov 02 22:25:53.341880 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.341844 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a621f1-jxbw5"] Nov 02 22:25:53.928354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.928327 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:25:53.981743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.981711 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-token\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-token\") pod \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " Nov 02 22:25:53.981924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.981855 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-data\") pod \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " Nov 02 22:25:53.981924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.981885 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-socket\") pod \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " Nov 02 22:25:53.981924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.981913 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-credential-socket\") pod \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " Nov 02 22:25:53.982290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.981965 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istiod-ca-cert\") pod \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " Nov 02 22:25:53.982290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.982175 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-certs\") pod \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " Nov 02 22:25:53.982290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.982232 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-envoy\") pod \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " Nov 02 22:25:53.982290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.982253 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-credential-socket" pod "adfc3202-ffd9-4470-b3bb-f0ced5d8311f" (UID: "adfc3202-ffd9-4470-b3bb-f0ced5d8311f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:53.982499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.982293 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-podinfo\") pod \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " Nov 02 22:25:53.982499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.982339 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-kube-api-access-h2lkb\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-kube-api-access-h2lkb\") pod \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\" (UID: \"adfc3202-ffd9-4470-b3bb-f0ced5d8311f\") " Nov 02 22:25:53.982499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.982461 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istiod-ca-cert" pod "adfc3202-ffd9-4470-b3bb-f0ced5d8311f" (UID: "adfc3202-ffd9-4470-b3bb-f0ced5d8311f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:25:53.982639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.982508 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-certs" pod "adfc3202-ffd9-4470-b3bb-f0ced5d8311f" (UID: "adfc3202-ffd9-4470-b3bb-f0ced5d8311f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:53.982811 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.982786 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-socket" pod "adfc3202-ffd9-4470-b3bb-f0ced5d8311f" (UID: "adfc3202-ffd9-4470-b3bb-f0ced5d8311f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:53.983075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.983040 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:53.983075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.983063 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:53.983075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.983079 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:53.983389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.983092 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:53.983470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.983389 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-data" pod "adfc3202-ffd9-4470-b3bb-f0ced5d8311f" (UID: "adfc3202-ffd9-4470-b3bb-f0ced5d8311f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:53.985065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.985038 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-podinfo" pod "adfc3202-ffd9-4470-b3bb-f0ced5d8311f" (UID: "adfc3202-ffd9-4470-b3bb-f0ced5d8311f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:25:53.985299 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.985264 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-envoy" pod "adfc3202-ffd9-4470-b3bb-f0ced5d8311f" (UID: "adfc3202-ffd9-4470-b3bb-f0ced5d8311f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:25:53.985501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.985477 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-kube-api-access-h2lkb" pod "adfc3202-ffd9-4470-b3bb-f0ced5d8311f" (UID: "adfc3202-ffd9-4470-b3bb-f0ced5d8311f"). InnerVolumeSpecName "kube-api-access-h2lkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:53.985746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:53.985725 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-token" pod "adfc3202-ffd9-4470-b3bb-f0ced5d8311f" (UID: "adfc3202-ffd9-4470-b3bb-f0ced5d8311f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:25:54.051941 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.051906 4309 generic.go:356] "Generic (PLEG): container finished" podID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerID="68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e" exitCode=0 Nov 02 22:25:54.052144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.051953 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" Nov 02 22:25:54.052144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.051985 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" event={"ID":"adfc3202-ffd9-4470-b3bb-f0ced5d8311f","Type":"ContainerDied","Data":"68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e"} Nov 02 22:25:54.052144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.052035 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww" event={"ID":"adfc3202-ffd9-4470-b3bb-f0ced5d8311f","Type":"ContainerDied","Data":"d9f17cebd6d15e187098ca32f57818c51225367294022f80e41ad06d1f5cb145"} Nov 02 22:25:54.052144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.052064 4309 scope.go:122] "RemoveContainer" containerID="68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e" Nov 02 22:25:54.065232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.065212 4309 scope.go:122] "RemoveContainer" containerID="68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e" Nov 02 22:25:54.065493 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:25:54.065470 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e\": container with ID starting with 68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e not found: ID does not exist" containerID="68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e" Nov 02 22:25:54.065570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.065504 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e"} err="failed to get container status \"68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e\": rpc error: code = NotFound desc = could not find container \"68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e\": container with ID starting with 68c88c05d638cf1b1935bc65aed295b32baec1d36b4082a4a8aaac2302ffa51e not found: ID does not exist" Nov 02 22:25:54.077717 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.077686 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww"] Nov 02 22:25:54.081409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.081387 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d28bd1-openshift-default-6957ccd87c-77hww"] Nov 02 22:25:54.084370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.084348 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:54.084500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.084372 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:54.084500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.084383 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:54.084500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.084391 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:54.084500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.084400 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2lkb\" (UniqueName: \"kubernetes.io/projected/adfc3202-ffd9-4470-b3bb-f0ced5d8311f-kube-api-access-h2lkb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:25:54.204077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.203994 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014ce160-7bc9-4b3b-9dc1-64a34697c3c7" path="/var/lib/kubelet/pods/014ce160-7bc9-4b3b-9dc1-64a34697c3c7/volumes" Nov 02 22:25:54.205379 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.205330 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" path="/var/lib/kubelet/pods/adfc3202-ffd9-4470-b3bb-f0ced5d8311f/volumes" Nov 02 22:25:54.390004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.389926 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" podUID="3a6a2ece-b673-4597-8a52-2b9bb819f81d" containerName="payload-processing" containerID="cri-o://66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379" gracePeriod=25 Nov 02 22:25:54.963156 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:54.963119 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" podUID="3a6a2ece-b673-4597-8a52-2b9bb819f81d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:25:57.211794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.202740 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4"] Nov 02 22:25:57.211794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.202967 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" containerID="cri-o://40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130" gracePeriod=30 Nov 02 22:25:57.637853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.637821 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6"] Nov 02 22:25:57.641949 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.641925 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7bb56e-kgds6"] Nov 02 22:25:57.906782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.906692 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz"] Nov 02 22:25:57.908069 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.908037 4309 cpu_manager.go:414] "containerMap: removing container" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" Nov 02 22:25:57.908355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.908336 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" Nov 02 22:25:57.908481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.908469 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7b867ca1-8d8e-45de-9124-0186a7d23373" containerName="revoke-keys" Nov 02 22:25:57.908572 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.908561 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b867ca1-8d8e-45de-9124-0186a7d23373" containerName="revoke-keys" Nov 02 22:25:57.908693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.908666 4309 cpu_manager.go:414] "containerMap: removing container" podUID="525c64d4-fb38-4bbd-bec8-42cdb184d7af" containerName="revoke-keys" Nov 02 22:25:57.908798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.908785 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525c64d4-fb38-4bbd-bec8-42cdb184d7af" containerName="revoke-keys" Nov 02 22:25:57.908895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.908884 4309 cpu_manager.go:414] "containerMap: removing container" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" Nov 02 22:25:57.909054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.909038 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" Nov 02 22:25:57.909440 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.909423 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="525c64d4-fb38-4bbd-bec8-42cdb184d7af" containerName="revoke-keys" Nov 02 22:25:57.909554 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.909541 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfc3202-ffd9-4470-b3bb-f0ced5d8311f" containerName="istio-proxy" Nov 02 22:25:57.909759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.909736 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" Nov 02 22:25:57.909879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.909857 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" Nov 02 22:25:57.909999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.909985 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b867ca1-8d8e-45de-9124-0186a7d23373" containerName="revoke-keys" Nov 02 22:25:57.927693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.925526 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz"] Nov 02 22:25:57.927693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.925664 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:57.935410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.935383 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:25:57.935553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.935481 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:25:57.935553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.935383 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:25:57.935553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:57.935494 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:25:58.012738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.012696 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-719012-wbblz\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.012901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.012869 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-719012-wbblz\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.013047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.013023 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/a5179704-a92f-4e88-9fe6-8984b96b235b-kube-api-access-8bfb8\") pod \"maas-api-revoke-keys-e2e-worker-w3-719012-wbblz\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.114272 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.114230 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/a5179704-a92f-4e88-9fe6-8984b96b235b-kube-api-access-8bfb8\") pod \"maas-api-revoke-keys-e2e-worker-w3-719012-wbblz\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.114480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.114396 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-719012-wbblz\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.114552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.114533 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-719012-wbblz\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.115138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.115113 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-719012-wbblz\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.116937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.116906 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-719012-wbblz\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.123786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.123762 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/a5179704-a92f-4e88-9fe6-8984b96b235b-kube-api-access-8bfb8\") pod \"maas-api-revoke-keys-e2e-worker-w3-719012-wbblz\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.190558 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.190486 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b867ca1-8d8e-45de-9124-0186a7d23373" path="/var/lib/kubelet/pods/7b867ca1-8d8e-45de-9124-0186a7d23373/volumes" Nov 02 22:25:58.247951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.247920 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:25:58.532148 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:25:58.532119 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5179704_a92f_4e88_9fe6_8984b96b235b.slice/crio-bd6f4b12057a90191ffcc883ebf7fd0b466595d0206f12e9ec3518137c7489fb WatchSource:0}: Error finding container bd6f4b12057a90191ffcc883ebf7fd0b466595d0206f12e9ec3518137c7489fb: Status 404 returned error can't find the container with id bd6f4b12057a90191ffcc883ebf7fd0b466595d0206f12e9ec3518137c7489fb Nov 02 22:25:58.532931 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:58.532903 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz"] Nov 02 22:25:59.097656 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:59.097550 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" event={"ID":"a5179704-a92f-4e88-9fe6-8984b96b235b","Type":"ContainerStarted","Data":"d0214a4f9d8c4605d483f156a398aad49583de483ec6fd89af896b228f1a7c91"} Nov 02 22:25:59.097656 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:59.097624 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" event={"ID":"a5179704-a92f-4e88-9fe6-8984b96b235b","Type":"ContainerStarted","Data":"bd6f4b12057a90191ffcc883ebf7fd0b466595d0206f12e9ec3518137c7489fb"} Nov 02 22:25:59.114363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:59.114308 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" podStartSLOduration=2.11429359 podStartE2EDuration="2.11429359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:25:59.112717578 +0000 UTC m=+1687.050830797" watchObservedRunningTime="2026-11-02 22:25:59.11429359 +0000 UTC m=+1687.052406808" Nov 02 22:25:59.240336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:59.240306 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2"] Nov 02 22:25:59.258441 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:59.258004 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s"] Nov 02 22:25:59.928241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:59.928207 4309 scope.go:122] "RemoveContainer" containerID="3e945989d54efdeee2da835d05379f8cb1ea322b4178cd412f3dc978778ddc67" Nov 02 22:25:59.996088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:25:59.996051 4309 scope.go:122] "RemoveContainer" containerID="d042a70ca66d57ac619f5a9a56fc03fb1f5f6b198e4e249578c0f913ee0f5324" Nov 02 22:26:00.010579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.010554 4309 scope.go:122] "RemoveContainer" containerID="743522f113eca95a76d09e2a4145f2c633911071646ac934e73d2193289f09b4" Nov 02 22:26:00.021258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.021229 4309 scope.go:122] "RemoveContainer" containerID="e6fa27d4cb66a7796f832a3f04abfcfdbadee80d3772ae5db853eb59476c3253" Nov 02 22:26:00.033878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.033854 4309 scope.go:122] "RemoveContainer" containerID="a135876985cc8feaef3f231309107e2abd066b3146ab689985aa77d0f4d6ad2c" Nov 02 22:26:00.044838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.044815 4309 scope.go:122] "RemoveContainer" containerID="7a150a935524d0671ee45c697fef1e6b4c19bd1a694fd24e191a0ac049cade77" Nov 02 22:26:00.054368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.054349 4309 scope.go:122] "RemoveContainer" containerID="e4849ba6806dd284a08f83695bd46ce7dce743412c794fd232731d73ef1d0d68" Nov 02 22:26:00.064588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.064561 4309 scope.go:122] "RemoveContainer" containerID="095388bc3637d785d51b111bdef55fc9dd1d8877cbe4ee3b0d611d5338990e11" Nov 02 22:26:00.109016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.108981 4309 generic.go:356] "Generic (PLEG): container finished" podID="a5179704-a92f-4e88-9fe6-8984b96b235b" containerID="d0214a4f9d8c4605d483f156a398aad49583de483ec6fd89af896b228f1a7c91" exitCode=0 Nov 02 22:26:00.109129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.109059 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" event={"ID":"a5179704-a92f-4e88-9fe6-8984b96b235b","Type":"ContainerDied","Data":"d0214a4f9d8c4605d483f156a398aad49583de483ec6fd89af896b228f1a7c91"} Nov 02 22:26:00.721712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.721662 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px"] Nov 02 22:26:00.722527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.722495 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" containerID="cri-o://eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98" gracePeriod=30 Nov 02 22:26:00.833695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.825478 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6"] Nov 02 22:26:00.833695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.826509 4309 cpu_manager.go:414] "containerMap: removing container" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" Nov 02 22:26:00.833695 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.826528 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5748639-a17c-4ea1-9400-ac7fd3519cc7" containerName="istio-proxy" Nov 02 22:26:00.850788 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.848937 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:00.856688 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.856645 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6"] Nov 02 22:26:00.949696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.947005 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tlqd\" (UniqueName: \"kubernetes.io/projected/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-kube-api-access-9tlqd\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:00.949696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.947114 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:00.949696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:00.947439 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:01.049248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.049158 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tlqd\" (UniqueName: \"kubernetes.io/projected/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-kube-api-access-9tlqd\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:01.049248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.049240 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:01.049500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.049409 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:01.049968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.049942 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:01.051960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.051938 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:01.057858 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.057828 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tlqd\" (UniqueName: \"kubernetes.io/projected/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-kube-api-access-9tlqd\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:01.171175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.171146 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:01.288625 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.288589 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:26:01.456227 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.456192 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-openshift-ca\") pod \"a5179704-a92f-4e88-9fe6-8984b96b235b\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " Nov 02 22:26:01.456409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.456292 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5179704-a92f-4e88-9fe6-8984b96b235b-kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/a5179704-a92f-4e88-9fe6-8984b96b235b-kube-api-access-8bfb8\") pod \"a5179704-a92f-4e88-9fe6-8984b96b235b\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " Nov 02 22:26:01.456553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.456535 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-xks-ca\") pod \"a5179704-a92f-4e88-9fe6-8984b96b235b\" (UID: \"a5179704-a92f-4e88-9fe6-8984b96b235b\") " Nov 02 22:26:01.456612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.456555 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-openshift-ca" pod "a5179704-a92f-4e88-9fe6-8984b96b235b" (UID: "a5179704-a92f-4e88-9fe6-8984b96b235b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:01.457260 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.457233 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:01.461830 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.461807 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-xks-ca" pod "a5179704-a92f-4e88-9fe6-8984b96b235b" (UID: "a5179704-a92f-4e88-9fe6-8984b96b235b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:01.461950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.461857 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5179704-a92f-4e88-9fe6-8984b96b235b-kube-api-access-8bfb8" pod "a5179704-a92f-4e88-9fe6-8984b96b235b" (UID: "a5179704-a92f-4e88-9fe6-8984b96b235b"). InnerVolumeSpecName "kube-api-access-8bfb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:01.476460 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:26:01.476435 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95d50f7_16c6_4d71_84a1_db66a7c38e2b.slice/crio-00826377eeebc426a72181beb216f8ca2a37328df75ce4aeecca74ce05e58bd5 WatchSource:0}: Error finding container 00826377eeebc426a72181beb216f8ca2a37328df75ce4aeecca74ce05e58bd5: Status 404 returned error can't find the container with id 00826377eeebc426a72181beb216f8ca2a37328df75ce4aeecca74ce05e58bd5 Nov 02 22:26:01.476947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.476922 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6"] Nov 02 22:26:01.559450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.559410 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5179704-a92f-4e88-9fe6-8984b96b235b-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:01.559450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:01.559453 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/a5179704-a92f-4e88-9fe6-8984b96b235b-kube-api-access-8bfb8\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:02.138050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.138018 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" Nov 02 22:26:02.138465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.138051 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz" event={"ID":"a5179704-a92f-4e88-9fe6-8984b96b235b","Type":"ContainerDied","Data":"bd6f4b12057a90191ffcc883ebf7fd0b466595d0206f12e9ec3518137c7489fb"} Nov 02 22:26:02.138465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.138101 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd6f4b12057a90191ffcc883ebf7fd0b466595d0206f12e9ec3518137c7489fb" Nov 02 22:26:02.140053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.140025 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" event={"ID":"e95d50f7-16c6-4d71-84a1-db66a7c38e2b","Type":"ContainerStarted","Data":"b7c28d5c2e9b736c1bb32e8e770fb62cbe27eb3d4105afbf3ae6ff56bfb01d98"} Nov 02 22:26:02.140194 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.140071 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" event={"ID":"e95d50f7-16c6-4d71-84a1-db66a7c38e2b","Type":"ContainerStarted","Data":"00826377eeebc426a72181beb216f8ca2a37328df75ce4aeecca74ce05e58bd5"} Nov 02 22:26:02.159763 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.159710 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" podStartSLOduration=2.159661959 podStartE2EDuration="2.159661959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:02.157371798 +0000 UTC m=+1690.095485018" watchObservedRunningTime="2026-11-02 22:26:02.159661959 +0000 UTC m=+1690.097775177" Nov 02 22:26:02.523104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.523074 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:26:02.573035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573000 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-certs\") pod \"04aeb037-5886-453a-84ea-9cd21986f34a\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " Nov 02 22:26:02.573234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573064 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-data\") pod \"04aeb037-5886-453a-84ea-9cd21986f34a\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " Nov 02 22:26:02.573234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573113 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-kube-api-access-d4d7g\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-kube-api-access-d4d7g\") pod \"04aeb037-5886-453a-84ea-9cd21986f34a\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " Nov 02 22:26:02.573234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573211 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-socket\") pod \"04aeb037-5886-453a-84ea-9cd21986f34a\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " Nov 02 22:26:02.573492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573270 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04aeb037-5886-453a-84ea-9cd21986f34a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aeb037-5886-453a-84ea-9cd21986f34a-istio-podinfo\") pod \"04aeb037-5886-453a-84ea-9cd21986f34a\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " Nov 02 22:26:02.573492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573310 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04aeb037-5886-453a-84ea-9cd21986f34a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aeb037-5886-453a-84ea-9cd21986f34a-istiod-ca-cert\") pod \"04aeb037-5886-453a-84ea-9cd21986f34a\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " Nov 02 22:26:02.573492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573357 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-istio-token\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-istio-token\") pod \"04aeb037-5886-453a-84ea-9cd21986f34a\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " Nov 02 22:26:02.573492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573442 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-envoy\") pod \"04aeb037-5886-453a-84ea-9cd21986f34a\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " Nov 02 22:26:02.573492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573467 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-data" pod "04aeb037-5886-453a-84ea-9cd21986f34a" (UID: "04aeb037-5886-453a-84ea-9cd21986f34a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:02.573903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573531 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-credential-socket\") pod \"04aeb037-5886-453a-84ea-9cd21986f34a\" (UID: \"04aeb037-5886-453a-84ea-9cd21986f34a\") " Nov 02 22:26:02.573903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.573751 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-certs" pod "04aeb037-5886-453a-84ea-9cd21986f34a" (UID: "04aeb037-5886-453a-84ea-9cd21986f34a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:02.574049 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.574022 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-credential-socket" pod "04aeb037-5886-453a-84ea-9cd21986f34a" (UID: "04aeb037-5886-453a-84ea-9cd21986f34a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:02.574422 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.574203 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:02.574422 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.574228 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:02.574422 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.574243 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:02.575285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.575065 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04aeb037-5886-453a-84ea-9cd21986f34a-istiod-ca-cert" pod "04aeb037-5886-453a-84ea-9cd21986f34a" (UID: "04aeb037-5886-453a-84ea-9cd21986f34a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:02.575481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.575437 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-socket" pod "04aeb037-5886-453a-84ea-9cd21986f34a" (UID: "04aeb037-5886-453a-84ea-9cd21986f34a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:02.576965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.576939 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-kube-api-access-d4d7g" pod "04aeb037-5886-453a-84ea-9cd21986f34a" (UID: "04aeb037-5886-453a-84ea-9cd21986f34a"). InnerVolumeSpecName "kube-api-access-d4d7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:02.578565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.578542 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-istio-token" pod "04aeb037-5886-453a-84ea-9cd21986f34a" (UID: "04aeb037-5886-453a-84ea-9cd21986f34a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:02.592087 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.592056 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-envoy" pod "04aeb037-5886-453a-84ea-9cd21986f34a" (UID: "04aeb037-5886-453a-84ea-9cd21986f34a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:02.595166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.595121 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04aeb037-5886-453a-84ea-9cd21986f34a-istio-podinfo" pod "04aeb037-5886-453a-84ea-9cd21986f34a" (UID: "04aeb037-5886-453a-84ea-9cd21986f34a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:02.653738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.652887 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc"] Nov 02 22:26:02.676295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.676257 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:02.676295 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.676299 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4d7g\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-kube-api-access-d4d7g\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:02.676493 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.676315 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04aeb037-5886-453a-84ea-9cd21986f34a-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:02.676493 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.676328 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04aeb037-5886-453a-84ea-9cd21986f34a-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:02.676493 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.676341 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04aeb037-5886-453a-84ea-9cd21986f34a-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:02.676493 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:02.676354 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04aeb037-5886-453a-84ea-9cd21986f34a-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:03.155050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.155007 4309 generic.go:356] "Generic (PLEG): container finished" podID="e95d50f7-16c6-4d71-84a1-db66a7c38e2b" containerID="b7c28d5c2e9b736c1bb32e8e770fb62cbe27eb3d4105afbf3ae6ff56bfb01d98" exitCode=0 Nov 02 22:26:03.155504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.155092 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" event={"ID":"e95d50f7-16c6-4d71-84a1-db66a7c38e2b","Type":"ContainerDied","Data":"b7c28d5c2e9b736c1bb32e8e770fb62cbe27eb3d4105afbf3ae6ff56bfb01d98"} Nov 02 22:26:03.157145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.157117 4309 generic.go:356] "Generic (PLEG): container finished" podID="04aeb037-5886-453a-84ea-9cd21986f34a" containerID="40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130" exitCode=0 Nov 02 22:26:03.157285 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.157264 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" event={"ID":"04aeb037-5886-453a-84ea-9cd21986f34a","Type":"ContainerDied","Data":"40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130"} Nov 02 22:26:03.157327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.157311 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" event={"ID":"04aeb037-5886-453a-84ea-9cd21986f34a","Type":"ContainerDied","Data":"66ecdefdbd54b32b9a9d7f9080d9ea795e28637f0b915b57fd323cd5e2f25dab"} Nov 02 22:26:03.157393 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.157343 4309 scope.go:122] "RemoveContainer" containerID="40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130" Nov 02 22:26:03.157489 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.157464 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4" Nov 02 22:26:03.170821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.170798 4309 scope.go:122] "RemoveContainer" containerID="40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130" Nov 02 22:26:03.171155 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:03.171128 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130\": container with ID starting with 40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130 not found: ID does not exist" containerID="40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130" Nov 02 22:26:03.171208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.171168 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130"} err="failed to get container status \"40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130\": rpc error: code = NotFound desc = could not find container \"40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130\": container with ID starting with 40bfc999c6e9e6f7e738e747216892d43d104fb302cff7867929d48ec1334130 not found: ID does not exist" Nov 02 22:26:03.189609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.189573 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4"] Nov 02 22:26:03.193737 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.193685 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-00c72a-openshift-default-7455c6bdcf-942z4"] Nov 02 22:26:03.635258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.635228 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p"] Nov 02 22:26:03.638996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:03.638972 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5eba37-xgj9p"] Nov 02 22:26:04.192389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.192350 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" path="/var/lib/kubelet/pods/04aeb037-5886-453a-84ea-9cd21986f34a/volumes" Nov 02 22:26:04.193058 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.193036 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525c64d4-fb38-4bbd-bec8-42cdb184d7af" path="/var/lib/kubelet/pods/525c64d4-fb38-4bbd-bec8-42cdb184d7af/volumes" Nov 02 22:26:04.241412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.241227 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" podUID="dbb5861e-35ce-4e87-adf8-a1e60203ee3e" containerName="payload-processing" containerID="cri-o://93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400" gracePeriod=25 Nov 02 22:26:04.259155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.259051 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" podUID="7db96d89-7a42-4efb-aad5-f2d19119b488" containerName="payload-pre-processing" containerID="cri-o://71da3f4205c4f8042aee8f59609c0f7c2e05ba77b9824dd984ca264a60507c47" gracePeriod=25 Nov 02 22:26:04.387856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.387828 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:04.409313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.409268 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-kube-api-access-9tlqd\" (UniqueName: \"kubernetes.io/projected/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-kube-api-access-9tlqd\") pod \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " Nov 02 22:26:04.409499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.409331 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-xks-ca\") pod \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " Nov 02 22:26:04.409499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.409453 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-openshift-ca\") pod \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\" (UID: \"e95d50f7-16c6-4d71-84a1-db66a7c38e2b\") " Nov 02 22:26:04.410420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.410393 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-openshift-ca" pod "e95d50f7-16c6-4d71-84a1-db66a7c38e2b" (UID: "e95d50f7-16c6-4d71-84a1-db66a7c38e2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:04.412162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.412134 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-xks-ca" pod "e95d50f7-16c6-4d71-84a1-db66a7c38e2b" (UID: "e95d50f7-16c6-4d71-84a1-db66a7c38e2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:04.412757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.412726 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-kube-api-access-9tlqd" pod "e95d50f7-16c6-4d71-84a1-db66a7c38e2b" (UID: "e95d50f7-16c6-4d71-84a1-db66a7c38e2b"). InnerVolumeSpecName "kube-api-access-9tlqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:04.511429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.511358 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:04.511429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.511390 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:04.511429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.511405 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tlqd\" (UniqueName: \"kubernetes.io/projected/e95d50f7-16c6-4d71-84a1-db66a7c38e2b-kube-api-access-9tlqd\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:04.548217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.548193 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:26:04.612844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.612771 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-plugins-config-volume\") pod \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\" (UID: \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\") " Nov 02 22:26:04.613256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.613223 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-plugins-config-volume" pod "dbb5861e-35ce-4e87-adf8-a1e60203ee3e" (UID: "dbb5861e-35ce-4e87-adf8-a1e60203ee3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:04.613403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.613311 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-kube-api-access-b4c2g\" (UniqueName: \"kubernetes.io/projected/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-kube-api-access-b4c2g\") pod \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\" (UID: \"dbb5861e-35ce-4e87-adf8-a1e60203ee3e\") " Nov 02 22:26:04.613946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.613923 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:04.615718 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.615666 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-kube-api-access-b4c2g" pod "dbb5861e-35ce-4e87-adf8-a1e60203ee3e" (UID: "dbb5861e-35ce-4e87-adf8-a1e60203ee3e"). InnerVolumeSpecName "kube-api-access-b4c2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:04.719641 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.717095 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4c2g\" (UniqueName: \"kubernetes.io/projected/dbb5861e-35ce-4e87-adf8-a1e60203ee3e-kube-api-access-b4c2g\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:04.861247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.861212 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c"] Nov 02 22:26:04.861479 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.861455 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" containerID="cri-o://fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f" gracePeriod=30 Nov 02 22:26:04.990768 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:04.986795 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" podUID="3a6a2ece-b673-4597-8a52-2b9bb819f81d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:05.015811 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.015767 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" podUID="7db96d89-7a42-4efb-aad5-f2d19119b488" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:05.173923 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.173897 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" Nov 02 22:26:05.174120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.173938 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6" event={"ID":"e95d50f7-16c6-4d71-84a1-db66a7c38e2b","Type":"ContainerDied","Data":"00826377eeebc426a72181beb216f8ca2a37328df75ce4aeecca74ce05e58bd5"} Nov 02 22:26:05.174120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.173986 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00826377eeebc426a72181beb216f8ca2a37328df75ce4aeecca74ce05e58bd5" Nov 02 22:26:05.175773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.175739 4309 generic.go:356] "Generic (PLEG): container finished" podID="dbb5861e-35ce-4e87-adf8-a1e60203ee3e" containerID="93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400" exitCode=0 Nov 02 22:26:05.175924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.175791 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" Nov 02 22:26:05.175924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.175812 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" event={"ID":"dbb5861e-35ce-4e87-adf8-a1e60203ee3e","Type":"ContainerDied","Data":"93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400"} Nov 02 22:26:05.175924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.175862 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2" event={"ID":"dbb5861e-35ce-4e87-adf8-a1e60203ee3e","Type":"ContainerDied","Data":"7be9650d3391b0805c4d79524195bcc466ae6d6de88a7a6e26060d9686c9a543"} Nov 02 22:26:05.175924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.175885 4309 scope.go:122] "RemoveContainer" containerID="93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400" Nov 02 22:26:05.190240 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.190218 4309 scope.go:122] "RemoveContainer" containerID="93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400" Nov 02 22:26:05.191379 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:05.191357 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400\": container with ID starting with 93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400 not found: ID does not exist" containerID="93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400" Nov 02 22:26:05.191482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.191386 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400"} err="failed to get container status \"93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400\": rpc error: code = NotFound desc = could not find container \"93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400\": container with ID starting with 93fda490e076226d5e7d3963b470a39b6dc37b4b9eeb5ee4ec0995224eb7b400 not found: ID does not exist" Nov 02 22:26:05.206201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.206170 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2"] Nov 02 22:26:05.210333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.210306 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5eba37-8cdcd489c-8tfl2"] Nov 02 22:26:05.482438 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.482400 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs"] Nov 02 22:26:05.483431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483396 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e95d50f7-16c6-4d71-84a1-db66a7c38e2b" containerName="revoke-keys" Nov 02 22:26:05.483431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483423 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95d50f7-16c6-4d71-84a1-db66a7c38e2b" containerName="revoke-keys" Nov 02 22:26:05.483612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483447 4309 cpu_manager.go:414] "containerMap: removing container" podUID="a5179704-a92f-4e88-9fe6-8984b96b235b" containerName="revoke-keys" Nov 02 22:26:05.483612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483455 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5179704-a92f-4e88-9fe6-8984b96b235b" containerName="revoke-keys" Nov 02 22:26:05.483612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483482 4309 cpu_manager.go:414] "containerMap: removing container" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" Nov 02 22:26:05.483612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483490 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" Nov 02 22:26:05.483612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483505 4309 cpu_manager.go:414] "containerMap: removing container" podUID="dbb5861e-35ce-4e87-adf8-a1e60203ee3e" containerName="payload-processing" Nov 02 22:26:05.483612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483516 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb5861e-35ce-4e87-adf8-a1e60203ee3e" containerName="payload-processing" Nov 02 22:26:05.483887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483659 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95d50f7-16c6-4d71-84a1-db66a7c38e2b" containerName="revoke-keys" Nov 02 22:26:05.483887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483687 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5179704-a92f-4e88-9fe6-8984b96b235b" containerName="revoke-keys" Nov 02 22:26:05.483887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483700 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="04aeb037-5886-453a-84ea-9cd21986f34a" containerName="istio-proxy" Nov 02 22:26:05.483887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.483711 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb5861e-35ce-4e87-adf8-a1e60203ee3e" containerName="payload-processing" Nov 02 22:26:05.510758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.504267 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.510758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.509511 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:26:05.510758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.510266 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:26:05.526759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.526224 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:26:05.526759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.526308 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:05.540725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.539264 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs"] Nov 02 22:26:05.548175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.547349 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzc98\" (UniqueName: \"kubernetes.io/projected/48d48b3c-1285-4739-b672-644d7c189a64-kube-api-access-lzc98\") pod \"maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.548175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.547510 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.548175 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.547587 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.649276 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.649243 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.649472 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.649298 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.649472 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.649430 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzc98\" (UniqueName: \"kubernetes.io/projected/48d48b3c-1285-4739-b672-644d7c189a64-kube-api-access-lzc98\") pod \"maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.649921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.649891 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.651633 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.651611 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.657074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.657051 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzc98\" (UniqueName: \"kubernetes.io/projected/48d48b3c-1285-4739-b672-644d7c189a64-kube-api-access-lzc98\") pod \"maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:05.858042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:05.858014 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:06.046181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.046150 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:26:06.155727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.155692 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-podinfo\") pod \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " Nov 02 22:26:06.155865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.155755 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-data\") pod \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " Nov 02 22:26:06.155913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.155883 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-credential-socket\") pod \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " Nov 02 22:26:06.155960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.155914 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-token\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-token\") pod \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " Nov 02 22:26:06.155960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.155952 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-certs\") pod \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " Nov 02 22:26:06.156074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156056 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-envoy\") pod \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " Nov 02 22:26:06.156144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156093 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istiod-ca-cert\") pod \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " Nov 02 22:26:06.156205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156140 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-kube-api-access-b72vk\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-kube-api-access-b72vk\") pod \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " Nov 02 22:26:06.156267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156225 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-credential-socket" pod "a8715498-8ba4-45f4-8214-fe3bd5c89fcb" (UID: "a8715498-8ba4-45f4-8214-fe3bd5c89fcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:06.156336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156295 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-socket\") pod \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\" (UID: \"a8715498-8ba4-45f4-8214-fe3bd5c89fcb\") " Nov 02 22:26:06.156447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156426 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-data" pod "a8715498-8ba4-45f4-8214-fe3bd5c89fcb" (UID: "a8715498-8ba4-45f4-8214-fe3bd5c89fcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:06.156646 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156625 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-certs" pod "a8715498-8ba4-45f4-8214-fe3bd5c89fcb" (UID: "a8715498-8ba4-45f4-8214-fe3bd5c89fcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:06.156762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156639 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs"] Nov 02 22:26:06.156888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156869 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-socket" pod "a8715498-8ba4-45f4-8214-fe3bd5c89fcb" (UID: "a8715498-8ba4-45f4-8214-fe3bd5c89fcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:06.156944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156890 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:06.156944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156913 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:06.156944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.156928 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:06.157191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.157154 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istiod-ca-cert" pod "a8715498-8ba4-45f4-8214-fe3bd5c89fcb" (UID: "a8715498-8ba4-45f4-8214-fe3bd5c89fcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:06.158317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.158219 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-podinfo" pod "a8715498-8ba4-45f4-8214-fe3bd5c89fcb" (UID: "a8715498-8ba4-45f4-8214-fe3bd5c89fcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:06.158833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.158802 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-kube-api-access-b72vk" pod "a8715498-8ba4-45f4-8214-fe3bd5c89fcb" (UID: "a8715498-8ba4-45f4-8214-fe3bd5c89fcb"). InnerVolumeSpecName "kube-api-access-b72vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:06.158999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.158977 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-token" pod "a8715498-8ba4-45f4-8214-fe3bd5c89fcb" (UID: "a8715498-8ba4-45f4-8214-fe3bd5c89fcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:06.159524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.159506 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-envoy" pod "a8715498-8ba4-45f4-8214-fe3bd5c89fcb" (UID: "a8715498-8ba4-45f4-8214-fe3bd5c89fcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:06.194669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.194634 4309 generic.go:356] "Generic (PLEG): container finished" podID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerID="eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98" exitCode=0 Nov 02 22:26:06.194832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.194760 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" Nov 02 22:26:06.196916 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.196887 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb5861e-35ce-4e87-adf8-a1e60203ee3e" path="/var/lib/kubelet/pods/dbb5861e-35ce-4e87-adf8-a1e60203ee3e/volumes" Nov 02 22:26:06.197644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.197615 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" event={"ID":"48d48b3c-1285-4739-b672-644d7c189a64","Type":"ContainerStarted","Data":"d13c414dc75c4c988bbd0e807e0971da6d47d68c76edb4128eb70d8297e6b913"} Nov 02 22:26:06.197845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.197666 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" event={"ID":"a8715498-8ba4-45f4-8214-fe3bd5c89fcb","Type":"ContainerDied","Data":"eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98"} Nov 02 22:26:06.197845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.197712 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px" event={"ID":"a8715498-8ba4-45f4-8214-fe3bd5c89fcb","Type":"ContainerDied","Data":"6565d6b2dc7a7f7b74655fe5050e2222a595003c0bbba11da26b64c97ffb6851"} Nov 02 22:26:06.197845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.197742 4309 scope.go:122] "RemoveContainer" containerID="eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98" Nov 02 22:26:06.212268 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.212247 4309 scope.go:122] "RemoveContainer" containerID="eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98" Nov 02 22:26:06.212621 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:06.212538 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98\": container with ID starting with eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98 not found: ID does not exist" containerID="eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98" Nov 02 22:26:06.212621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.212565 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98"} err="failed to get container status \"eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98\": rpc error: code = NotFound desc = could not find container \"eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98\": container with ID starting with eac47d4b56f855094b0bd883b05eeea9df3033be7518fe0b2a6eee9bf7f3ba98 not found: ID does not exist" Nov 02 22:26:06.225567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.225528 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px"] Nov 02 22:26:06.229088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.229067 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-nc2px"] Nov 02 22:26:06.258730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.258699 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:06.258730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.258724 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:06.258730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.258734 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:06.258994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.258743 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:06.258994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.258753 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:06.258994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:06.258764 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b72vk\" (UniqueName: \"kubernetes.io/projected/a8715498-8ba4-45f4-8214-fe3bd5c89fcb-kube-api-access-b72vk\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:07.205499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:07.205454 4309 generic.go:356] "Generic (PLEG): container finished" podID="48d48b3c-1285-4739-b672-644d7c189a64" containerID="7325174642f4b3b473592e4389d4b33f423148c8151ab1e62bdcf3243309a92f" exitCode=0 Nov 02 22:26:07.205741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:07.205581 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" event={"ID":"48d48b3c-1285-4739-b672-644d7c189a64","Type":"ContainerDied","Data":"7325174642f4b3b473592e4389d4b33f423148c8151ab1e62bdcf3243309a92f"} Nov 02 22:26:08.041659 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.041618 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph"] Nov 02 22:26:08.191712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.191653 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" path="/var/lib/kubelet/pods/a8715498-8ba4-45f4-8214-fe3bd5c89fcb/volumes" Nov 02 22:26:08.544591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.544552 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:08.619499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.619453 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-xks-ca\") pod \"48d48b3c-1285-4739-b672-644d7c189a64\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " Nov 02 22:26:08.619499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.619505 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-openshift-ca\") pod \"48d48b3c-1285-4739-b672-644d7c189a64\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " Nov 02 22:26:08.619790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.619665 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d48b3c-1285-4739-b672-644d7c189a64-kube-api-access-lzc98\" (UniqueName: \"kubernetes.io/projected/48d48b3c-1285-4739-b672-644d7c189a64-kube-api-access-lzc98\") pod \"48d48b3c-1285-4739-b672-644d7c189a64\" (UID: \"48d48b3c-1285-4739-b672-644d7c189a64\") " Nov 02 22:26:08.619912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.619884 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-openshift-ca" pod "48d48b3c-1285-4739-b672-644d7c189a64" (UID: "48d48b3c-1285-4739-b672-644d7c189a64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:08.620154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.620136 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:08.621711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.621683 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-xks-ca" pod "48d48b3c-1285-4739-b672-644d7c189a64" (UID: "48d48b3c-1285-4739-b672-644d7c189a64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:08.622243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.622226 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d48b3c-1285-4739-b672-644d7c189a64-kube-api-access-lzc98" pod "48d48b3c-1285-4739-b672-644d7c189a64" (UID: "48d48b3c-1285-4739-b672-644d7c189a64"). InnerVolumeSpecName "kube-api-access-lzc98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:08.721367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.721320 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzc98\" (UniqueName: \"kubernetes.io/projected/48d48b3c-1285-4739-b672-644d7c189a64-kube-api-access-lzc98\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:08.721367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:08.721364 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d48b3c-1285-4739-b672-644d7c189a64-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:09.225567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:09.225514 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" event={"ID":"48d48b3c-1285-4739-b672-644d7c189a64","Type":"ContainerDied","Data":"d13c414dc75c4c988bbd0e807e0971da6d47d68c76edb4128eb70d8297e6b913"} Nov 02 22:26:09.226383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:09.225578 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d13c414dc75c4c988bbd0e807e0971da6d47d68c76edb4128eb70d8297e6b913" Nov 02 22:26:09.226383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:09.225642 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs" Nov 02 22:26:10.155131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.155103 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:26:10.235239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.235209 4309 generic.go:356] "Generic (PLEG): container finished" podID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerID="fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f" exitCode=0 Nov 02 22:26:10.235779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.235259 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" Nov 02 22:26:10.235779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.235294 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" event={"ID":"82133f4f-73f3-42a8-8bb9-02673f70fe28","Type":"ContainerDied","Data":"fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f"} Nov 02 22:26:10.235779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.235354 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c" event={"ID":"82133f4f-73f3-42a8-8bb9-02673f70fe28","Type":"ContainerDied","Data":"764dc7255925d04e047e5dca30160ed82822bea6ec45abd61b8191fcf2bf1c98"} Nov 02 22:26:10.235779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.235379 4309 scope.go:122] "RemoveContainer" containerID="fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f" Nov 02 22:26:10.245131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245104 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-certs\") pod \"82133f4f-73f3-42a8-8bb9-02673f70fe28\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " Nov 02 22:26:10.245263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245179 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-data\") pod \"82133f4f-73f3-42a8-8bb9-02673f70fe28\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " Nov 02 22:26:10.245263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245217 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-envoy\") pod \"82133f4f-73f3-42a8-8bb9-02673f70fe28\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " Nov 02 22:26:10.245382 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245265 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-kube-api-access-f2xtl\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-kube-api-access-f2xtl\") pod \"82133f4f-73f3-42a8-8bb9-02673f70fe28\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " Nov 02 22:26:10.245444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245404 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-certs" pod "82133f4f-73f3-42a8-8bb9-02673f70fe28" (UID: "82133f4f-73f3-42a8-8bb9-02673f70fe28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:10.245444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245416 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-podinfo\") pod \"82133f4f-73f3-42a8-8bb9-02673f70fe28\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " Nov 02 22:26:10.245560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245442 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-data" pod "82133f4f-73f3-42a8-8bb9-02673f70fe28" (UID: "82133f4f-73f3-42a8-8bb9-02673f70fe28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:10.245560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245462 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82133f4f-73f3-42a8-8bb9-02673f70fe28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82133f4f-73f3-42a8-8bb9-02673f70fe28-istiod-ca-cert\") pod \"82133f4f-73f3-42a8-8bb9-02673f70fe28\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " Nov 02 22:26:10.245560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245496 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-socket\") pod \"82133f4f-73f3-42a8-8bb9-02673f70fe28\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " Nov 02 22:26:10.245824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245596 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-token\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-token\") pod \"82133f4f-73f3-42a8-8bb9-02673f70fe28\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " Nov 02 22:26:10.245824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245632 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-credential-socket\") pod \"82133f4f-73f3-42a8-8bb9-02673f70fe28\" (UID: \"82133f4f-73f3-42a8-8bb9-02673f70fe28\") " Nov 02 22:26:10.245824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245786 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-socket" pod "82133f4f-73f3-42a8-8bb9-02673f70fe28" (UID: "82133f4f-73f3-42a8-8bb9-02673f70fe28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:10.246001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.245845 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82133f4f-73f3-42a8-8bb9-02673f70fe28-istiod-ca-cert" pod "82133f4f-73f3-42a8-8bb9-02673f70fe28" (UID: "82133f4f-73f3-42a8-8bb9-02673f70fe28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:10.246176 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.246153 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-credential-socket" pod "82133f4f-73f3-42a8-8bb9-02673f70fe28" (UID: "82133f4f-73f3-42a8-8bb9-02673f70fe28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:10.246654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.246607 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82133f4f-73f3-42a8-8bb9-02673f70fe28-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:10.246764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.246746 4309 scope.go:122] "RemoveContainer" containerID="fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f" Nov 02 22:26:10.246852 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.246835 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:10.246926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.246860 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:10.246926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.246877 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:10.246926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.246893 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:10.247051 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:10.247027 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f\": container with ID starting with fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f not found: ID does not exist" containerID="fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f" Nov 02 22:26:10.247109 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.247048 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f"} err="failed to get container status \"fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f\": rpc error: code = NotFound desc = could not find container \"fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f\": container with ID starting with fc0cddfb2db22307a0e907651c0c77814b111ebeebadf8ffd313fe5f39452d1f not found: ID does not exist" Nov 02 22:26:10.248000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.247977 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-podinfo" pod "82133f4f-73f3-42a8-8bb9-02673f70fe28" (UID: "82133f4f-73f3-42a8-8bb9-02673f70fe28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:10.248000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.247987 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-envoy" pod "82133f4f-73f3-42a8-8bb9-02673f70fe28" (UID: "82133f4f-73f3-42a8-8bb9-02673f70fe28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:10.248157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.248115 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-kube-api-access-f2xtl" pod "82133f4f-73f3-42a8-8bb9-02673f70fe28" (UID: "82133f4f-73f3-42a8-8bb9-02673f70fe28"). InnerVolumeSpecName "kube-api-access-f2xtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:10.248157 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.248142 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-token" pod "82133f4f-73f3-42a8-8bb9-02673f70fe28" (UID: "82133f4f-73f3-42a8-8bb9-02673f70fe28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:10.313242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.313200 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l"] Nov 02 22:26:10.332522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.332488 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb"] Nov 02 22:26:10.348263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.348236 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:10.348263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.348262 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2xtl\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-kube-api-access-f2xtl\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:10.348452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.348274 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:10.348452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.348283 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82133f4f-73f3-42a8-8bb9-02673f70fe28-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:10.371575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.371543 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf"] Nov 02 22:26:10.372244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.372227 4309 cpu_manager.go:414] "containerMap: removing container" podUID="48d48b3c-1285-4739-b672-644d7c189a64" containerName="revoke-keys" Nov 02 22:26:10.372301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.372246 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d48b3c-1285-4739-b672-644d7c189a64" containerName="revoke-keys" Nov 02 22:26:10.372301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.372291 4309 cpu_manager.go:414] "containerMap: removing container" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" Nov 02 22:26:10.372301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.372296 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" Nov 02 22:26:10.372403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.372304 4309 cpu_manager.go:414] "containerMap: removing container" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" Nov 02 22:26:10.372403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.372309 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" Nov 02 22:26:10.372403 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.372395 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d48b3c-1285-4739-b672-644d7c189a64" containerName="revoke-keys" Nov 02 22:26:10.372505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.372411 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" containerName="istio-proxy" Nov 02 22:26:10.372505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.372416 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8715498-8ba4-45f4-8214-fe3bd5c89fcb" containerName="istio-proxy" Nov 02 22:26:10.377347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.377329 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.383022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.382955 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w"] Nov 02 22:26:10.385244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.385223 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:26:10.385917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.385857 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:10.386268 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.386020 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:26:10.386268 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.386030 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:26:10.387926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.387906 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf"] Nov 02 22:26:10.388042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.387997 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.412045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.412009 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w"] Nov 02 22:26:10.449969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.449940 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfdd38af-47snf\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.449969 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.449972 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngxp\" (UniqueName: \"kubernetes.io/projected/c3b702b3-84ec-462b-b7f7-c406b9a162de-kube-api-access-zngxp\") pod \"maas-api-revoke-sub-bfdd38af-47snf\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.450200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.450166 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfdd38af-47snf\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.450292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.450279 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvx49\" (UniqueName: \"kubernetes.io/projected/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-kube-api-access-kvx49\") pod \"maas-api-revoke-sub-1d87d388-p852w\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.450336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.450321 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d87d388-p852w\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.450382 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.450338 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d87d388-p852w\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.555699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.553183 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d87d388-p852w\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.555699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.553238 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d87d388-p852w\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.555699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.553323 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfdd38af-47snf\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.555699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.553350 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngxp\" (UniqueName: \"kubernetes.io/projected/c3b702b3-84ec-462b-b7f7-c406b9a162de-kube-api-access-zngxp\") pod \"maas-api-revoke-sub-bfdd38af-47snf\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.555699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.553481 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfdd38af-47snf\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.555699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.553634 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvx49\" (UniqueName: \"kubernetes.io/projected/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-kube-api-access-kvx49\") pod \"maas-api-revoke-sub-1d87d388-p852w\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.555699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.555497 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfdd38af-47snf\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.571089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.556320 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d87d388-p852w\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.571089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.558707 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfdd38af-47snf\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.571089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.566129 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d87d388-p852w\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.574587 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.574492 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvx49\" (UniqueName: \"kubernetes.io/projected/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-kube-api-access-kvx49\") pod \"maas-api-revoke-sub-1d87d388-p852w\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.575256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.575229 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngxp\" (UniqueName: \"kubernetes.io/projected/c3b702b3-84ec-462b-b7f7-c406b9a162de-kube-api-access-zngxp\") pod \"maas-api-revoke-sub-bfdd38af-47snf\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.615274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.615241 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c"] Nov 02 22:26:10.621482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.621454 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7bb56e-openshift-default-558658cfff-gmx7c"] Nov 02 22:26:10.722231 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.722144 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:10.732536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.732505 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:10.846420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.845506 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz"] Nov 02 22:26:10.846420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:10.845730 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" containerID="cri-o://398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4" gracePeriod=30 Nov 02 22:26:11.050262 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.031928 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf"] Nov 02 22:26:11.101662 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.099371 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w"] Nov 02 22:26:11.124874 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:26:11.124840 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f13b1b4_ef53_45cb_bbf4_a7c9b9e42bf8.slice/crio-5cdc7e3ab4fefb75716c327b11f60e1c2d28503f25dde9d211d351b02a769757 WatchSource:0}: Error finding container 5cdc7e3ab4fefb75716c327b11f60e1c2d28503f25dde9d211d351b02a769757: Status 404 returned error can't find the container with id 5cdc7e3ab4fefb75716c327b11f60e1c2d28503f25dde9d211d351b02a769757 Nov 02 22:26:11.277712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.245918 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" event={"ID":"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8","Type":"ContainerStarted","Data":"5cdc7e3ab4fefb75716c327b11f60e1c2d28503f25dde9d211d351b02a769757"} Nov 02 22:26:11.278347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.278293 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" event={"ID":"c3b702b3-84ec-462b-b7f7-c406b9a162de","Type":"ContainerStarted","Data":"8da35d836fd7c449d5d54b1fc090481344079dc3ec360735b528765b271abfb8"} Nov 02 22:26:11.437954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.437919 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52"] Nov 02 22:26:11.443666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.443647 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.451283 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.451257 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52"] Nov 02 22:26:11.598774 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.598738 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwfk\" (UniqueName: \"kubernetes.io/projected/48074a11-e38e-4b34-a9da-db3fcfe95a84-kube-api-access-4lwfk\") pod \"maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.598892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.598828 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.598892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.598872 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.606891 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:11.606862 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:26:11.700944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.700915 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwfk\" (UniqueName: \"kubernetes.io/projected/48074a11-e38e-4b34-a9da-db3fcfe95a84-kube-api-access-4lwfk\") pod \"maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.701084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.700991 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.701084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.701031 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.701870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.701847 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.703206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.703185 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.708229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.708206 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwfk\" (UniqueName: \"kubernetes.io/projected/48074a11-e38e-4b34-a9da-db3fcfe95a84-kube-api-access-4lwfk\") pod \"maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:11.761310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:11.761273 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:12.056860 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:26:12.056829 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48074a11_e38e_4b34_a9da_db3fcfe95a84.slice/crio-f25c4f44a0ec247ce4b99597d6f6303ba48c08049b144e5735ae59ce6bc99409 WatchSource:0}: Error finding container f25c4f44a0ec247ce4b99597d6f6303ba48c08049b144e5735ae59ce6bc99409: Status 404 returned error can't find the container with id f25c4f44a0ec247ce4b99597d6f6303ba48c08049b144e5735ae59ce6bc99409 Nov 02 22:26:12.057334 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:12.057311 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52"] Nov 02 22:26:12.192772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:12.192743 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82133f4f-73f3-42a8-8bb9-02673f70fe28" path="/var/lib/kubelet/pods/82133f4f-73f3-42a8-8bb9-02673f70fe28/volumes" Nov 02 22:26:12.294749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:12.289650 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" event={"ID":"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8","Type":"ContainerStarted","Data":"fd24133c356812ec04684cdab238b8b7d34fbb8abe367ac98f60522f18dc529a"} Nov 02 22:26:12.294749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:12.291573 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" event={"ID":"c3b702b3-84ec-462b-b7f7-c406b9a162de","Type":"ContainerStarted","Data":"c401da5c9200c63e7ad6f48ca2fef2047bcd84905701ed3c242b4add650a6a66"} Nov 02 22:26:12.294749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:12.293488 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" event={"ID":"48074a11-e38e-4b34-a9da-db3fcfe95a84","Type":"ContainerStarted","Data":"f25c4f44a0ec247ce4b99597d6f6303ba48c08049b144e5735ae59ce6bc99409"} Nov 02 22:26:12.316130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:12.316071 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" podStartSLOduration=2.316054417 podStartE2EDuration="2.316054417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:12.311183143 +0000 UTC m=+1700.249296361" watchObservedRunningTime="2026-11-02 22:26:12.316054417 +0000 UTC m=+1700.254167635" Nov 02 22:26:12.327593 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:12.327535 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" podStartSLOduration=2.327521971 podStartE2EDuration="2.327521971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:12.325308725 +0000 UTC m=+1700.263421943" watchObservedRunningTime="2026-11-02 22:26:12.327521971 +0000 UTC m=+1700.265635188" Nov 02 22:26:13.042667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:13.042603 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" podUID="7315bae5-39b1-4241-bba0-303c9a5ad5e7" containerName="payload-processing" containerID="cri-o://e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b" gracePeriod=25 Nov 02 22:26:13.304534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:13.304432 4309 generic.go:356] "Generic (PLEG): container finished" podID="0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8" containerID="fd24133c356812ec04684cdab238b8b7d34fbb8abe367ac98f60522f18dc529a" exitCode=0 Nov 02 22:26:13.305019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:13.304521 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" event={"ID":"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8","Type":"ContainerDied","Data":"fd24133c356812ec04684cdab238b8b7d34fbb8abe367ac98f60522f18dc529a"} Nov 02 22:26:13.306372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:13.306335 4309 generic.go:356] "Generic (PLEG): container finished" podID="c3b702b3-84ec-462b-b7f7-c406b9a162de" containerID="c401da5c9200c63e7ad6f48ca2fef2047bcd84905701ed3c242b4add650a6a66" exitCode=0 Nov 02 22:26:13.306503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:13.306424 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" event={"ID":"c3b702b3-84ec-462b-b7f7-c406b9a162de","Type":"ContainerDied","Data":"c401da5c9200c63e7ad6f48ca2fef2047bcd84905701ed3c242b4add650a6a66"} Nov 02 22:26:13.308225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:13.308200 4309 generic.go:356] "Generic (PLEG): container finished" podID="48074a11-e38e-4b34-a9da-db3fcfe95a84" containerID="e3d4069f8ecf89fbbee220fb783c702c5ceb0bd929995ad0b3e1547f76ea1415" exitCode=0 Nov 02 22:26:13.308352 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:13.308238 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" event={"ID":"48074a11-e38e-4b34-a9da-db3fcfe95a84","Type":"ContainerDied","Data":"e3d4069f8ecf89fbbee220fb783c702c5ceb0bd929995ad0b3e1547f76ea1415"} Nov 02 22:26:13.563511 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:13.563482 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6"] Nov 02 22:26:13.567141 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:13.567114 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-ac0910-b5mz6"] Nov 02 22:26:14.189441 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.189406 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95d50f7-16c6-4d71-84a1-db66a7c38e2b" path="/var/lib/kubelet/pods/e95d50f7-16c6-4d71-84a1-db66a7c38e2b/volumes" Nov 02 22:26:14.382411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.382153 4309 patch_prober.go:29] interesting pod/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:26:14.382411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.382200 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:26:14.576051 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.576023 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:14.586375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.586355 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:14.591846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.591814 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:14.724920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.724871 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:26:14.749910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.749846 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-openshift-ca\") pod \"c3b702b3-84ec-462b-b7f7-c406b9a162de\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " Nov 02 22:26:14.749910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.749882 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-openshift-ca\") pod \"48074a11-e38e-4b34-a9da-db3fcfe95a84\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " Nov 02 22:26:14.750138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750113 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-xks-ca\") pod \"48074a11-e38e-4b34-a9da-db3fcfe95a84\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " Nov 02 22:26:14.750252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750235 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-kube-api-access-kvx49\" (UniqueName: \"kubernetes.io/projected/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-kube-api-access-kvx49\") pod \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " Nov 02 22:26:14.750342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750246 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-openshift-ca" pod "48074a11-e38e-4b34-a9da-db3fcfe95a84" (UID: "48074a11-e38e-4b34-a9da-db3fcfe95a84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:14.750342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750267 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-openshift-ca" pod "c3b702b3-84ec-462b-b7f7-c406b9a162de" (UID: "c3b702b3-84ec-462b-b7f7-c406b9a162de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:14.750342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750286 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-openshift-ca\") pod \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " Nov 02 22:26:14.750342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750314 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-xks-ca\") pod \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\" (UID: \"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8\") " Nov 02 22:26:14.750544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750347 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b702b3-84ec-462b-b7f7-c406b9a162de-kube-api-access-zngxp\" (UniqueName: \"kubernetes.io/projected/c3b702b3-84ec-462b-b7f7-c406b9a162de-kube-api-access-zngxp\") pod \"c3b702b3-84ec-462b-b7f7-c406b9a162de\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " Nov 02 22:26:14.750544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750424 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48074a11-e38e-4b34-a9da-db3fcfe95a84-kube-api-access-4lwfk\" (UniqueName: \"kubernetes.io/projected/48074a11-e38e-4b34-a9da-db3fcfe95a84-kube-api-access-4lwfk\") pod \"48074a11-e38e-4b34-a9da-db3fcfe95a84\" (UID: \"48074a11-e38e-4b34-a9da-db3fcfe95a84\") " Nov 02 22:26:14.750544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750507 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-xks-ca\") pod \"c3b702b3-84ec-462b-b7f7-c406b9a162de\" (UID: \"c3b702b3-84ec-462b-b7f7-c406b9a162de\") " Nov 02 22:26:14.750751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.750666 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-openshift-ca" pod "0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8" (UID: "0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:14.752716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.752341 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.752716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.752382 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.752716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.752399 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.753561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.753515 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-xks-ca" pod "c3b702b3-84ec-462b-b7f7-c406b9a162de" (UID: "c3b702b3-84ec-462b-b7f7-c406b9a162de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:14.754637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.754617 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-kube-api-access-kvx49" pod "0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8" (UID: "0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8"). InnerVolumeSpecName "kube-api-access-kvx49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:14.754756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.754652 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b702b3-84ec-462b-b7f7-c406b9a162de-kube-api-access-zngxp" pod "c3b702b3-84ec-462b-b7f7-c406b9a162de" (UID: "c3b702b3-84ec-462b-b7f7-c406b9a162de"). InnerVolumeSpecName "kube-api-access-zngxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:14.754756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.754750 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-xks-ca" pod "48074a11-e38e-4b34-a9da-db3fcfe95a84" (UID: "48074a11-e38e-4b34-a9da-db3fcfe95a84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:14.754846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.754773 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-xks-ca" pod "0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8" (UID: "0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:14.756170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.756143 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48074a11-e38e-4b34-a9da-db3fcfe95a84-kube-api-access-4lwfk" pod "48074a11-e38e-4b34-a9da-db3fcfe95a84" (UID: "48074a11-e38e-4b34-a9da-db3fcfe95a84"). InnerVolumeSpecName "kube-api-access-4lwfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:14.853423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.853138 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6a2ece-b673-4597-8a52-2b9bb819f81d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a6a2ece-b673-4597-8a52-2b9bb819f81d-plugins-config-volume\") pod \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\" (UID: \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\") " Nov 02 22:26:14.853423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.853299 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6a2ece-b673-4597-8a52-2b9bb819f81d-kube-api-access-5lpnr\" (UniqueName: \"kubernetes.io/projected/3a6a2ece-b673-4597-8a52-2b9bb819f81d-kube-api-access-5lpnr\") pod \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\" (UID: \"3a6a2ece-b673-4597-8a52-2b9bb819f81d\") " Nov 02 22:26:14.854088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.853925 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48074a11-e38e-4b34-a9da-db3fcfe95a84-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.854088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.853956 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvx49\" (UniqueName: \"kubernetes.io/projected/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-kube-api-access-kvx49\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.854088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.853972 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.854088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.853985 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngxp\" (UniqueName: \"kubernetes.io/projected/c3b702b3-84ec-462b-b7f7-c406b9a162de-kube-api-access-zngxp\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.854088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.853998 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lwfk\" (UniqueName: \"kubernetes.io/projected/48074a11-e38e-4b34-a9da-db3fcfe95a84-kube-api-access-4lwfk\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.854088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.854011 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b702b3-84ec-462b-b7f7-c406b9a162de-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.873712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.865331 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6a2ece-b673-4597-8a52-2b9bb819f81d-plugins-config-volume" pod "3a6a2ece-b673-4597-8a52-2b9bb819f81d" (UID: "3a6a2ece-b673-4597-8a52-2b9bb819f81d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:14.885202 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.885144 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6a2ece-b673-4597-8a52-2b9bb819f81d-kube-api-access-5lpnr" pod "3a6a2ece-b673-4597-8a52-2b9bb819f81d" (UID: "3a6a2ece-b673-4597-8a52-2b9bb819f81d"). InnerVolumeSpecName "kube-api-access-5lpnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:14.955640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.955601 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lpnr\" (UniqueName: \"kubernetes.io/projected/3a6a2ece-b673-4597-8a52-2b9bb819f81d-kube-api-access-5lpnr\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:14.955640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:14.955636 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a6a2ece-b673-4597-8a52-2b9bb819f81d-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:15.004019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.003934 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" podUID="7db96d89-7a42-4efb-aad5-f2d19119b488" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:15.327574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.327542 4309 generic.go:356] "Generic (PLEG): container finished" podID="3a6a2ece-b673-4597-8a52-2b9bb819f81d" containerID="66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379" exitCode=0 Nov 02 22:26:15.327794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.327628 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" event={"ID":"3a6a2ece-b673-4597-8a52-2b9bb819f81d","Type":"ContainerDied","Data":"66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379"} Nov 02 22:26:15.327794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.327654 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" Nov 02 22:26:15.327794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.327716 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj" event={"ID":"3a6a2ece-b673-4597-8a52-2b9bb819f81d","Type":"ContainerDied","Data":"81bf15e1413d36f37f4e2e959dd48bc6cd6961b51253ac453b297f29df665ec6"} Nov 02 22:26:15.327794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.327748 4309 scope.go:122] "RemoveContainer" containerID="66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379" Nov 02 22:26:15.329266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.329124 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" Nov 02 22:26:15.329266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.329187 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52" event={"ID":"48074a11-e38e-4b34-a9da-db3fcfe95a84","Type":"ContainerDied","Data":"f25c4f44a0ec247ce4b99597d6f6303ba48c08049b144e5735ae59ce6bc99409"} Nov 02 22:26:15.329266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.329230 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f25c4f44a0ec247ce4b99597d6f6303ba48c08049b144e5735ae59ce6bc99409" Nov 02 22:26:15.331968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.331950 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" Nov 02 22:26:15.332092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.331979 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w" event={"ID":"0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8","Type":"ContainerDied","Data":"5cdc7e3ab4fefb75716c327b11f60e1c2d28503f25dde9d211d351b02a769757"} Nov 02 22:26:15.332092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.332024 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cdc7e3ab4fefb75716c327b11f60e1c2d28503f25dde9d211d351b02a769757" Nov 02 22:26:15.333709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.333630 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" event={"ID":"c3b702b3-84ec-462b-b7f7-c406b9a162de","Type":"ContainerDied","Data":"8da35d836fd7c449d5d54b1fc090481344079dc3ec360735b528765b271abfb8"} Nov 02 22:26:15.333709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.333704 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8da35d836fd7c449d5d54b1fc090481344079dc3ec360735b528765b271abfb8" Nov 02 22:26:15.333873 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.333760 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf" Nov 02 22:26:15.341907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.341886 4309 scope.go:122] "RemoveContainer" containerID="66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379" Nov 02 22:26:15.342195 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:15.342175 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379\": container with ID starting with 66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379 not found: ID does not exist" containerID="66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379" Nov 02 22:26:15.342275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.342207 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379"} err="failed to get container status \"66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379\": rpc error: code = NotFound desc = could not find container \"66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379\": container with ID starting with 66a6753b13e306de9e64b60ae8a290276dee53d2fd78bc090ee8863cd330e379 not found: ID does not exist" Nov 02 22:26:15.366130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.366105 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj"] Nov 02 22:26:15.371275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.371253 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a621f1-56fc9f6bc7-42zfj"] Nov 02 22:26:15.668357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:15.668267 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4"] Nov 02 22:26:16.132730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.132692 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" podUID="7315bae5-39b1-4241-bba0-303c9a5ad5e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:16.191228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.191196 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a6a2ece-b673-4597-8a52-2b9bb819f81d" path="/var/lib/kubelet/pods/3a6a2ece-b673-4597-8a52-2b9bb819f81d/volumes" Nov 02 22:26:16.196474 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.196452 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:26:16.198597 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.198578 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-envoy\") pod \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " Nov 02 22:26:16.198713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.198644 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-socket\") pod \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " Nov 02 22:26:16.198713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.198696 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-certs\") pod \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " Nov 02 22:26:16.198814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.198742 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-kube-api-access-4hvz7\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-kube-api-access-4hvz7\") pod \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " Nov 02 22:26:16.198914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.198893 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-data\") pod \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " Nov 02 22:26:16.198989 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.198905 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-socket" pod "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" (UID: "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:16.199045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199000 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-credential-socket\") pod \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " Nov 02 22:26:16.199045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199016 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-certs" pod "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" (UID: "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:16.199145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199112 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-podinfo\") pod \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " Nov 02 22:26:16.199208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199131 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-data" pod "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" (UID: "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:16.199208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199154 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-token\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-token\") pod \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " Nov 02 22:26:16.199208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199185 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-credential-socket" pod "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" (UID: "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:16.199397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199208 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istiod-ca-cert\") pod \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\" (UID: \"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54\") " Nov 02 22:26:16.199607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199582 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istiod-ca-cert" pod "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" (UID: "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:16.199897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199878 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:16.199984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199902 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:16.199984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199918 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:16.199984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199932 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:16.199984 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.199945 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:16.201221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.201200 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-token" pod "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" (UID: "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:16.201578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.201558 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-podinfo" pod "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" (UID: "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:16.201655 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.201628 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-envoy" pod "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" (UID: "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:16.202060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.202034 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-kube-api-access-4hvz7" pod "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" (UID: "f2efa0fa-4f83-4432-a568-a5d2ff6d6b54"). InnerVolumeSpecName "kube-api-access-4hvz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:16.301781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.301554 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:16.301781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.301590 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:16.301781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.301604 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:16.301781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.301619 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hvz7\" (UniqueName: \"kubernetes.io/projected/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54-kube-api-access-4hvz7\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:16.309234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.309201 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89"] Nov 02 22:26:16.347429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.347397 4309 generic.go:356] "Generic (PLEG): container finished" podID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerID="398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4" exitCode=0 Nov 02 22:26:16.347628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.347445 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" Nov 02 22:26:16.347628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.347488 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" event={"ID":"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54","Type":"ContainerDied","Data":"398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4"} Nov 02 22:26:16.347628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.347547 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz" event={"ID":"f2efa0fa-4f83-4432-a568-a5d2ff6d6b54","Type":"ContainerDied","Data":"d3b60527b623af737542b8172da059bf8284102919ac15001f5036b2baded3a6"} Nov 02 22:26:16.347628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.347579 4309 scope.go:122] "RemoveContainer" containerID="398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4" Nov 02 22:26:16.358829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.358805 4309 scope.go:122] "RemoveContainer" containerID="c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742" Nov 02 22:26:16.368154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.368137 4309 scope.go:122] "RemoveContainer" containerID="398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4" Nov 02 22:26:16.368499 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:16.368456 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4\": container with ID starting with 398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4 not found: ID does not exist" containerID="398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4" Nov 02 22:26:16.368499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.368490 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4"} err="failed to get container status \"398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4\": rpc error: code = NotFound desc = could not find container \"398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4\": container with ID starting with 398859057e35aea00e50a4a9ffc41b844eb06615d15ca6675a6f0ed13381bce4 not found: ID does not exist" Nov 02 22:26:16.368730 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.368508 4309 scope.go:122] "RemoveContainer" containerID="c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742" Nov 02 22:26:16.369613 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:16.368860 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742\": container with ID starting with c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742 not found: ID does not exist" containerID="c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742" Nov 02 22:26:16.369613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.368892 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742"} err="failed to get container status \"c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742\": rpc error: code = NotFound desc = could not find container \"c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742\": container with ID starting with c7d5e8e4de09f0335e1157659d3a6215e5c7605e544e1be1ab74aa76e9cfc742 not found: ID does not exist" Nov 02 22:26:16.373391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.373368 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz"] Nov 02 22:26:16.378524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:16.378504 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5eba37-openshift-default-54c9b56b7f-pp6xz"] Nov 02 22:26:17.682784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:17.682739 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" containerID="cri-o://77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d" gracePeriod=45 Nov 02 22:26:17.974083 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:17.974056 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:26:18.020327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.019145 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-tmp-dir\") pod \"0127a148-aec0-4908-a7c5-c8027bc5b451\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " Nov 02 22:26:18.020327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.019200 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0127a148-aec0-4908-a7c5-c8027bc5b451-kube-api-access-9f2wm\" (UniqueName: \"kubernetes.io/projected/0127a148-aec0-4908-a7c5-c8027bc5b451-kube-api-access-9f2wm\") pod \"0127a148-aec0-4908-a7c5-c8027bc5b451\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " Nov 02 22:26:18.020327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.019270 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0127a148-aec0-4908-a7c5-c8027bc5b451-tls-certs\" (UniqueName: \"kubernetes.io/secret/0127a148-aec0-4908-a7c5-c8027bc5b451-tls-certs\") pod \"0127a148-aec0-4908-a7c5-c8027bc5b451\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " Nov 02 22:26:18.020327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.019309 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-model-cache\") pod \"0127a148-aec0-4908-a7c5-c8027bc5b451\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " Nov 02 22:26:18.020327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.019402 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-home\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-home\") pod \"0127a148-aec0-4908-a7c5-c8027bc5b451\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " Nov 02 22:26:18.020327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.019496 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-dshm\") pod \"0127a148-aec0-4908-a7c5-c8027bc5b451\" (UID: \"0127a148-aec0-4908-a7c5-c8027bc5b451\") " Nov 02 22:26:18.021292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.021210 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-tmp-dir" pod "0127a148-aec0-4908-a7c5-c8027bc5b451" (UID: "0127a148-aec0-4908-a7c5-c8027bc5b451"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:18.021292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.021228 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-home" pod "0127a148-aec0-4908-a7c5-c8027bc5b451" (UID: "0127a148-aec0-4908-a7c5-c8027bc5b451"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:18.021570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.021434 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-model-cache" pod "0127a148-aec0-4908-a7c5-c8027bc5b451" (UID: "0127a148-aec0-4908-a7c5-c8027bc5b451"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:18.021833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.021813 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:18.021955 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.021942 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:18.022064 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.022050 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:18.023183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.023153 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-dshm" pod "0127a148-aec0-4908-a7c5-c8027bc5b451" (UID: "0127a148-aec0-4908-a7c5-c8027bc5b451"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:18.024574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.024538 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0127a148-aec0-4908-a7c5-c8027bc5b451-tls-certs" pod "0127a148-aec0-4908-a7c5-c8027bc5b451" (UID: "0127a148-aec0-4908-a7c5-c8027bc5b451"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:18.024807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.024770 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0127a148-aec0-4908-a7c5-c8027bc5b451-kube-api-access-9f2wm" pod "0127a148-aec0-4908-a7c5-c8027bc5b451" (UID: "0127a148-aec0-4908-a7c5-c8027bc5b451"). InnerVolumeSpecName "kube-api-access-9f2wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:18.124105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.124067 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f2wm\" (UniqueName: \"kubernetes.io/projected/0127a148-aec0-4908-a7c5-c8027bc5b451-kube-api-access-9f2wm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:18.124105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.124109 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0127a148-aec0-4908-a7c5-c8027bc5b451-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:18.124328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.124127 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0127a148-aec0-4908-a7c5-c8027bc5b451-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:18.128046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.128014 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg"] Nov 02 22:26:18.128956 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.128935 4309 cpu_manager.go:414] "containerMap: removing container" podUID="0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8" containerName="revoke-keys" Nov 02 22:26:18.129019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.128959 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8" containerName="revoke-keys" Nov 02 22:26:18.129019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.128983 4309 cpu_manager.go:414] "containerMap: removing container" podUID="c3b702b3-84ec-462b-b7f7-c406b9a162de" containerName="revoke-keys" Nov 02 22:26:18.129019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.128991 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b702b3-84ec-462b-b7f7-c406b9a162de" containerName="revoke-keys" Nov 02 22:26:18.129019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129013 4309 cpu_manager.go:414] "containerMap: removing container" podUID="3a6a2ece-b673-4597-8a52-2b9bb819f81d" containerName="payload-processing" Nov 02 22:26:18.129147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129022 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6a2ece-b673-4597-8a52-2b9bb819f81d" containerName="payload-processing" Nov 02 22:26:18.129147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129042 4309 cpu_manager.go:414] "containerMap: removing container" podUID="48074a11-e38e-4b34-a9da-db3fcfe95a84" containerName="revoke-keys" Nov 02 22:26:18.129147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129049 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48074a11-e38e-4b34-a9da-db3fcfe95a84" containerName="revoke-keys" Nov 02 22:26:18.129147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129083 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" Nov 02 22:26:18.129147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129090 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" Nov 02 22:26:18.129147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129106 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" Nov 02 22:26:18.129147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129113 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" Nov 02 22:26:18.129147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129128 4309 cpu_manager.go:414] "containerMap: removing container" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" Nov 02 22:26:18.129147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129136 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" Nov 02 22:26:18.129417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129264 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8" containerName="revoke-keys" Nov 02 22:26:18.129417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129276 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b702b3-84ec-462b-b7f7-c406b9a162de" containerName="revoke-keys" Nov 02 22:26:18.129417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129288 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6a2ece-b673-4597-8a52-2b9bb819f81d" containerName="payload-processing" Nov 02 22:26:18.129417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129300 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="48074a11-e38e-4b34-a9da-db3fcfe95a84" containerName="revoke-keys" Nov 02 22:26:18.129417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129313 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" Nov 02 22:26:18.129417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129323 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" containerName="istio-proxy" Nov 02 22:26:18.129417 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.129333 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerName="main" Nov 02 22:26:18.133387 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.133369 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8"] Nov 02 22:26:18.133501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.133485 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.141821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.139010 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:18.141821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.139126 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:26:18.141821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.139202 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:26:18.141821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.139286 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:26:18.142095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.141764 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg"] Nov 02 22:26:18.142095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.141981 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.150844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.147061 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8"] Nov 02 22:26:18.190365 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.190295 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2efa0fa-4f83-4432-a568-a5d2ff6d6b54" path="/var/lib/kubelet/pods/f2efa0fa-4f83-4432-a568-a5d2ff6d6b54/volumes" Nov 02 22:26:18.225390 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.225344 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59w5\" (UniqueName: \"kubernetes.io/projected/08a108f3-2eba-48e1-b96b-a6022d20c72c-kube-api-access-x59w5\") pod \"maas-api-revoke-sub-755fa9e9-tfjsg\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.225562 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.225486 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84c3190-pttf8\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.225562 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.225523 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755fa9e9-tfjsg\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.225809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.225793 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84c3190-pttf8\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.225893 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.225845 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755fa9e9-tfjsg\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.226020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.226003 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzc7n\" (UniqueName: \"kubernetes.io/projected/1bcf78e8-6820-435c-9b82-6d29a9dab312-kube-api-access-wzc7n\") pod \"maas-api-revoke-sub-f84c3190-pttf8\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.326937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.326897 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59w5\" (UniqueName: \"kubernetes.io/projected/08a108f3-2eba-48e1-b96b-a6022d20c72c-kube-api-access-x59w5\") pod \"maas-api-revoke-sub-755fa9e9-tfjsg\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.327150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.326989 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84c3190-pttf8\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.327150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.327018 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755fa9e9-tfjsg\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.327150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.327130 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84c3190-pttf8\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.327320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.327178 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755fa9e9-tfjsg\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.327320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.327231 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzc7n\" (UniqueName: \"kubernetes.io/projected/1bcf78e8-6820-435c-9b82-6d29a9dab312-kube-api-access-wzc7n\") pod \"maas-api-revoke-sub-f84c3190-pttf8\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.327951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.327924 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-755fa9e9-tfjsg\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.328086 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.327926 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84c3190-pttf8\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.329769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.329744 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-755fa9e9-tfjsg\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.329899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.329857 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84c3190-pttf8\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.335397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.335377 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzc7n\" (UniqueName: \"kubernetes.io/projected/1bcf78e8-6820-435c-9b82-6d29a9dab312-kube-api-access-wzc7n\") pod \"maas-api-revoke-sub-f84c3190-pttf8\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.335740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.335722 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59w5\" (UniqueName: \"kubernetes.io/projected/08a108f3-2eba-48e1-b96b-a6022d20c72c-kube-api-access-x59w5\") pod \"maas-api-revoke-sub-755fa9e9-tfjsg\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.366428 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.366401 4309 generic.go:356] "Generic (PLEG): container finished" podID="0127a148-aec0-4908-a7c5-c8027bc5b451" containerID="77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d" exitCode=0 Nov 02 22:26:18.366543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.366449 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" Nov 02 22:26:18.366543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.366484 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" event={"ID":"0127a148-aec0-4908-a7c5-c8027bc5b451","Type":"ContainerDied","Data":"77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d"} Nov 02 22:26:18.366543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.366531 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc" event={"ID":"0127a148-aec0-4908-a7c5-c8027bc5b451","Type":"ContainerDied","Data":"5229d4785338a4f2bb7f21adb18f41eee21f49825fa6b720d35f4b05ba268058"} Nov 02 22:26:18.366654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.366550 4309 scope.go:122] "RemoveContainer" containerID="77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d" Nov 02 22:26:18.376799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.376774 4309 scope.go:122] "RemoveContainer" containerID="77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d" Nov 02 22:26:18.377071 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:18.377051 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d\": container with ID starting with 77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d not found: ID does not exist" containerID="77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d" Nov 02 22:26:18.377130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.377082 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d"} err="failed to get container status \"77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d\": rpc error: code = NotFound desc = could not find container \"77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d\": container with ID starting with 77ea47b3971c9609012f041cade3b0a9f0b191f232460aa9e91f87c0ffe30e8d not found: ID does not exist" Nov 02 22:26:18.386434 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.386409 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc"] Nov 02 22:26:18.390026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.390004 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-719012/premium-w3-719012-kserve-67cfd8c5cc-97xnc"] Nov 02 22:26:18.465377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.463234 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:18.467802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.467770 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:18.771533 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.771491 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg"] Nov 02 22:26:18.772047 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:26:18.772024 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a108f3_2eba_48e1_b96b_a6022d20c72c.slice/crio-75bc971776cc56a552ad64267dca7281de40569902831e16a87b6daa1403e3d7 WatchSource:0}: Error finding container 75bc971776cc56a552ad64267dca7281de40569902831e16a87b6daa1403e3d7: Status 404 returned error can't find the container with id 75bc971776cc56a552ad64267dca7281de40569902831e16a87b6daa1403e3d7 Nov 02 22:26:18.778060 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:26:18.778031 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcf78e8_6820_435c_9b82_6d29a9dab312.slice/crio-521f3e155effc255cd0d9579ff106a8072a48394d33a00d5de06503b6661a20c WatchSource:0}: Error finding container 521f3e155effc255cd0d9579ff106a8072a48394d33a00d5de06503b6661a20c: Status 404 returned error can't find the container with id 521f3e155effc255cd0d9579ff106a8072a48394d33a00d5de06503b6661a20c Nov 02 22:26:18.778612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:18.778589 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8"] Nov 02 22:26:19.375454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:19.375354 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" event={"ID":"1bcf78e8-6820-435c-9b82-6d29a9dab312","Type":"ContainerStarted","Data":"04baf2c2e945aaeb7b96f582f3dbbec981f6fc2d591819c6451e211b74b5ddc0"} Nov 02 22:26:19.375454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:19.375420 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" event={"ID":"1bcf78e8-6820-435c-9b82-6d29a9dab312","Type":"ContainerStarted","Data":"521f3e155effc255cd0d9579ff106a8072a48394d33a00d5de06503b6661a20c"} Nov 02 22:26:19.377115 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:19.377089 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" event={"ID":"08a108f3-2eba-48e1-b96b-a6022d20c72c","Type":"ContainerStarted","Data":"9645bb9ffd851022f7ec215fda7d688a87c2a297e0ae8dbf835c6f3cf413bc4a"} Nov 02 22:26:19.377238 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:19.377130 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" event={"ID":"08a108f3-2eba-48e1-b96b-a6022d20c72c","Type":"ContainerStarted","Data":"75bc971776cc56a552ad64267dca7281de40569902831e16a87b6daa1403e3d7"} Nov 02 22:26:19.396601 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:19.396548 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" podStartSLOduration=1.3965335890000001 podStartE2EDuration="1.396533589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:19.391581183 +0000 UTC m=+1707.329694402" watchObservedRunningTime="2026-11-02 22:26:19.396533589 +0000 UTC m=+1707.334646809" Nov 02 22:26:19.407778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:19.407738 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" podStartSLOduration=1.407725826 podStartE2EDuration="1.407725826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:19.405451289 +0000 UTC m=+1707.343564508" watchObservedRunningTime="2026-11-02 22:26:19.407725826 +0000 UTC m=+1707.345839044" Nov 02 22:26:20.189386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:20.189343 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0127a148-aec0-4908-a7c5-c8027bc5b451" path="/var/lib/kubelet/pods/0127a148-aec0-4908-a7c5-c8027bc5b451/volumes" Nov 02 22:26:20.388605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:20.388558 4309 generic.go:356] "Generic (PLEG): container finished" podID="08a108f3-2eba-48e1-b96b-a6022d20c72c" containerID="9645bb9ffd851022f7ec215fda7d688a87c2a297e0ae8dbf835c6f3cf413bc4a" exitCode=0 Nov 02 22:26:20.388795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:20.388643 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" event={"ID":"08a108f3-2eba-48e1-b96b-a6022d20c72c","Type":"ContainerDied","Data":"9645bb9ffd851022f7ec215fda7d688a87c2a297e0ae8dbf835c6f3cf413bc4a"} Nov 02 22:26:20.390548 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:20.390525 4309 generic.go:356] "Generic (PLEG): container finished" podID="1bcf78e8-6820-435c-9b82-6d29a9dab312" containerID="04baf2c2e945aaeb7b96f582f3dbbec981f6fc2d591819c6451e211b74b5ddc0" exitCode=0 Nov 02 22:26:20.390659 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:20.390557 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" event={"ID":"1bcf78e8-6820-435c-9b82-6d29a9dab312","Type":"ContainerDied","Data":"04baf2c2e945aaeb7b96f582f3dbbec981f6fc2d591819c6451e211b74b5ddc0"} Nov 02 22:26:20.669482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:20.669439 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" podUID="2304b2a9-889a-4265-9ff3-25ac877f2f0d" containerName="payload-pre-processing" containerID="cri-o://7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7" gracePeriod=25 Nov 02 22:26:21.580973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.580650 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:21.627014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.626987 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:21.681148 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.681066 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcf78e8-6820-435c-9b82-6d29a9dab312-kube-api-access-wzc7n\" (UniqueName: \"kubernetes.io/projected/1bcf78e8-6820-435c-9b82-6d29a9dab312-kube-api-access-wzc7n\") pod \"1bcf78e8-6820-435c-9b82-6d29a9dab312\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " Nov 02 22:26:21.681148 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.681106 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-xks-ca\") pod \"08a108f3-2eba-48e1-b96b-a6022d20c72c\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " Nov 02 22:26:21.681384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.681165 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a108f3-2eba-48e1-b96b-a6022d20c72c-kube-api-access-x59w5\" (UniqueName: \"kubernetes.io/projected/08a108f3-2eba-48e1-b96b-a6022d20c72c-kube-api-access-x59w5\") pod \"08a108f3-2eba-48e1-b96b-a6022d20c72c\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " Nov 02 22:26:21.681384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.681331 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-xks-ca\") pod \"1bcf78e8-6820-435c-9b82-6d29a9dab312\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " Nov 02 22:26:21.681507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.681411 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-openshift-ca\") pod \"1bcf78e8-6820-435c-9b82-6d29a9dab312\" (UID: \"1bcf78e8-6820-435c-9b82-6d29a9dab312\") " Nov 02 22:26:21.681582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.681525 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-openshift-ca\") pod \"08a108f3-2eba-48e1-b96b-a6022d20c72c\" (UID: \"08a108f3-2eba-48e1-b96b-a6022d20c72c\") " Nov 02 22:26:21.683079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.682947 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-openshift-ca" pod "08a108f3-2eba-48e1-b96b-a6022d20c72c" (UID: "08a108f3-2eba-48e1-b96b-a6022d20c72c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:21.687722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.687649 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-openshift-ca" pod "1bcf78e8-6820-435c-9b82-6d29a9dab312" (UID: "1bcf78e8-6820-435c-9b82-6d29a9dab312"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:21.692406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.691096 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-xks-ca" pod "08a108f3-2eba-48e1-b96b-a6022d20c72c" (UID: "08a108f3-2eba-48e1-b96b-a6022d20c72c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:21.692406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.692078 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2"] Nov 02 22:26:21.693076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.693050 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-xks-ca" pod "1bcf78e8-6820-435c-9b82-6d29a9dab312" (UID: "1bcf78e8-6820-435c-9b82-6d29a9dab312"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:21.693165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.693135 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a108f3-2eba-48e1-b96b-a6022d20c72c-kube-api-access-x59w5" pod "08a108f3-2eba-48e1-b96b-a6022d20c72c" (UID: "08a108f3-2eba-48e1-b96b-a6022d20c72c"). InnerVolumeSpecName "kube-api-access-x59w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:21.693327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.693303 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcf78e8-6820-435c-9b82-6d29a9dab312-kube-api-access-wzc7n" pod "1bcf78e8-6820-435c-9b82-6d29a9dab312" (UID: "1bcf78e8-6820-435c-9b82-6d29a9dab312"). InnerVolumeSpecName "kube-api-access-wzc7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:21.705878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.705843 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5"] Nov 02 22:26:21.784478 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.784445 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzc7n\" (UniqueName: \"kubernetes.io/projected/1bcf78e8-6820-435c-9b82-6d29a9dab312-kube-api-access-wzc7n\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.784478 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.784479 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.784783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.784492 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x59w5\" (UniqueName: \"kubernetes.io/projected/08a108f3-2eba-48e1-b96b-a6022d20c72c-kube-api-access-x59w5\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.784783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.784506 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.784783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.784517 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcf78e8-6820-435c-9b82-6d29a9dab312-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:21.784783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:21.784529 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a108f3-2eba-48e1-b96b-a6022d20c72c-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:22.411793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.411756 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" event={"ID":"08a108f3-2eba-48e1-b96b-a6022d20c72c","Type":"ContainerDied","Data":"75bc971776cc56a552ad64267dca7281de40569902831e16a87b6daa1403e3d7"} Nov 02 22:26:22.412013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.411839 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75bc971776cc56a552ad64267dca7281de40569902831e16a87b6daa1403e3d7" Nov 02 22:26:22.412013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.411770 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg" Nov 02 22:26:22.413512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.413493 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" Nov 02 22:26:22.413512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.413494 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8" event={"ID":"1bcf78e8-6820-435c-9b82-6d29a9dab312","Type":"ContainerDied","Data":"521f3e155effc255cd0d9579ff106a8072a48394d33a00d5de06503b6661a20c"} Nov 02 22:26:22.413692 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.413555 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="521f3e155effc255cd0d9579ff106a8072a48394d33a00d5de06503b6661a20c" Nov 02 22:26:22.638447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.638416 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf"] Nov 02 22:26:22.639048 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.639034 4309 cpu_manager.go:414] "containerMap: removing container" podUID="1bcf78e8-6820-435c-9b82-6d29a9dab312" containerName="revoke-keys" Nov 02 22:26:22.639104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.639050 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcf78e8-6820-435c-9b82-6d29a9dab312" containerName="revoke-keys" Nov 02 22:26:22.639104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.639067 4309 cpu_manager.go:414] "containerMap: removing container" podUID="08a108f3-2eba-48e1-b96b-a6022d20c72c" containerName="revoke-keys" Nov 02 22:26:22.639104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.639074 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a108f3-2eba-48e1-b96b-a6022d20c72c" containerName="revoke-keys" Nov 02 22:26:22.639211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.639154 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcf78e8-6820-435c-9b82-6d29a9dab312" containerName="revoke-keys" Nov 02 22:26:22.639211 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.639166 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a108f3-2eba-48e1-b96b-a6022d20c72c" containerName="revoke-keys" Nov 02 22:26:22.643203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.643176 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.653256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.653228 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf"] Nov 02 22:26:22.657552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.657518 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz"] Nov 02 22:26:22.662419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.662389 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.675360 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.675331 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:26:22.675497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.675427 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:26:22.678508 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.678484 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz"] Nov 02 22:26:22.699426 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.699396 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22aa40e9-mgqdf\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.699592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.699433 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c821917-pwgmz\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.699592 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.699583 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhll2\" (UniqueName: \"kubernetes.io/projected/896a2b60-040c-44f1-b84c-d6fac267b653-kube-api-access-fhll2\") pod \"maas-api-revoke-sub-22aa40e9-mgqdf\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.699762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.699611 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7mjw\" (UniqueName: \"kubernetes.io/projected/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-kube-api-access-t7mjw\") pod \"maas-api-revoke-sub-5c821917-pwgmz\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.699762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.699648 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22aa40e9-mgqdf\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.699957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.699805 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c821917-pwgmz\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.740383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.740354 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:22.740547 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.740499 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:26:22.801368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.801335 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhll2\" (UniqueName: \"kubernetes.io/projected/896a2b60-040c-44f1-b84c-d6fac267b653-kube-api-access-fhll2\") pod \"maas-api-revoke-sub-22aa40e9-mgqdf\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.801575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.801384 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7mjw\" (UniqueName: \"kubernetes.io/projected/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-kube-api-access-t7mjw\") pod \"maas-api-revoke-sub-5c821917-pwgmz\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.801575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.801441 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22aa40e9-mgqdf\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.801575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.801517 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c821917-pwgmz\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.801852 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.801623 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22aa40e9-mgqdf\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.801852 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.801652 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c821917-pwgmz\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.802363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.802333 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22aa40e9-mgqdf\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.802486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.802384 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c821917-pwgmz\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.804095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.804072 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c821917-pwgmz\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.804468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.804449 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22aa40e9-mgqdf\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.809716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.809693 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhll2\" (UniqueName: \"kubernetes.io/projected/896a2b60-040c-44f1-b84c-d6fac267b653-kube-api-access-fhll2\") pod \"maas-api-revoke-sub-22aa40e9-mgqdf\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.809839 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.809764 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7mjw\" (UniqueName: \"kubernetes.io/projected/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-kube-api-access-t7mjw\") pod \"maas-api-revoke-sub-5c821917-pwgmz\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:22.993703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.992439 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:22.993703 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:22.993159 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:23.283100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:23.283070 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf"] Nov 02 22:26:23.283388 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:26:23.283359 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896a2b60_040c_44f1_b84c_d6fac267b653.slice/crio-a01a23354e94ca452abe2caf87e9d75ec45b3a3d365aee5e920e1b26a96cfd45 WatchSource:0}: Error finding container a01a23354e94ca452abe2caf87e9d75ec45b3a3d365aee5e920e1b26a96cfd45: Status 404 returned error can't find the container with id a01a23354e94ca452abe2caf87e9d75ec45b3a3d365aee5e920e1b26a96cfd45 Nov 02 22:26:23.313110 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:26:23.313077 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a04793_1d76_4d00_9f00_7d7c48d6a3ac.slice/crio-5946c751dfaa551e3a7f1b9deee1449289692aa14bccc14d2832885c4e0439ec WatchSource:0}: Error finding container 5946c751dfaa551e3a7f1b9deee1449289692aa14bccc14d2832885c4e0439ec: Status 404 returned error can't find the container with id 5946c751dfaa551e3a7f1b9deee1449289692aa14bccc14d2832885c4e0439ec Nov 02 22:26:23.313242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:23.313129 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz"] Nov 02 22:26:23.422521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:23.422484 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" event={"ID":"92a04793-1d76-4d00-9f00-7d7c48d6a3ac","Type":"ContainerStarted","Data":"5946c751dfaa551e3a7f1b9deee1449289692aa14bccc14d2832885c4e0439ec"} Nov 02 22:26:23.424309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:23.424276 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" event={"ID":"896a2b60-040c-44f1-b84c-d6fac267b653","Type":"ContainerStarted","Data":"a01a23354e94ca452abe2caf87e9d75ec45b3a3d365aee5e920e1b26a96cfd45"} Nov 02 22:26:24.030748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.030624 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz"] Nov 02 22:26:24.038849 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.038783 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-719012-wbblz"] Nov 02 22:26:24.192888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.192846 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5179704-a92f-4e88-9fe6-8984b96b235b" path="/var/lib/kubelet/pods/a5179704-a92f-4e88-9fe6-8984b96b235b/volumes" Nov 02 22:26:24.435776 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.435745 4309 generic.go:356] "Generic (PLEG): container finished" podID="7db96d89-7a42-4efb-aad5-f2d19119b488" containerID="71da3f4205c4f8042aee8f59609c0f7c2e05ba77b9824dd984ca264a60507c47" exitCode=0 Nov 02 22:26:24.435920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.435827 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" event={"ID":"7db96d89-7a42-4efb-aad5-f2d19119b488","Type":"ContainerDied","Data":"71da3f4205c4f8042aee8f59609c0f7c2e05ba77b9824dd984ca264a60507c47"} Nov 02 22:26:24.437724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.437661 4309 generic.go:356] "Generic (PLEG): container finished" podID="92a04793-1d76-4d00-9f00-7d7c48d6a3ac" containerID="ec1f13486e06b755d50742d8a158d642f7afc426500c2a1a9cc8cb00f39a4675" exitCode=0 Nov 02 22:26:24.437851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.437721 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" event={"ID":"92a04793-1d76-4d00-9f00-7d7c48d6a3ac","Type":"ContainerDied","Data":"ec1f13486e06b755d50742d8a158d642f7afc426500c2a1a9cc8cb00f39a4675"} Nov 02 22:26:24.440020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.439832 4309 generic.go:356] "Generic (PLEG): container finished" podID="896a2b60-040c-44f1-b84c-d6fac267b653" containerID="30e5c7728d03fd872b5756e93a5111abd5c399b23ae5f557355a3b542d83f68e" exitCode=0 Nov 02 22:26:24.440020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.439905 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" event={"ID":"896a2b60-040c-44f1-b84c-d6fac267b653","Type":"ContainerDied","Data":"30e5c7728d03fd872b5756e93a5111abd5c399b23ae5f557355a3b542d83f68e"} Nov 02 22:26:24.546081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.546052 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:26:24.630845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.630768 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db96d89-7a42-4efb-aad5-f2d19119b488-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db96d89-7a42-4efb-aad5-f2d19119b488-plugins-config-volume\") pod \"7db96d89-7a42-4efb-aad5-f2d19119b488\" (UID: \"7db96d89-7a42-4efb-aad5-f2d19119b488\") " Nov 02 22:26:24.631053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.630929 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db96d89-7a42-4efb-aad5-f2d19119b488-kube-api-access-stjzk\" (UniqueName: \"kubernetes.io/projected/7db96d89-7a42-4efb-aad5-f2d19119b488-kube-api-access-stjzk\") pod \"7db96d89-7a42-4efb-aad5-f2d19119b488\" (UID: \"7db96d89-7a42-4efb-aad5-f2d19119b488\") " Nov 02 22:26:24.631282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.631254 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db96d89-7a42-4efb-aad5-f2d19119b488-plugins-config-volume" pod "7db96d89-7a42-4efb-aad5-f2d19119b488" (UID: "7db96d89-7a42-4efb-aad5-f2d19119b488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:24.631506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.631487 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7db96d89-7a42-4efb-aad5-f2d19119b488-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:24.633182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.633156 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db96d89-7a42-4efb-aad5-f2d19119b488-kube-api-access-stjzk" pod "7db96d89-7a42-4efb-aad5-f2d19119b488" (UID: "7db96d89-7a42-4efb-aad5-f2d19119b488"). InnerVolumeSpecName "kube-api-access-stjzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:24.732466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:24.732430 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stjzk\" (UniqueName: \"kubernetes.io/projected/7db96d89-7a42-4efb-aad5-f2d19119b488-kube-api-access-stjzk\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.345979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.345932 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" containerID="cri-o://0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba" gracePeriod=45 Nov 02 22:26:25.354324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.354283 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" containerID="cri-o://9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c" gracePeriod=45 Nov 02 22:26:25.449623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.449595 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" Nov 02 22:26:25.449771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.449591 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s" event={"ID":"7db96d89-7a42-4efb-aad5-f2d19119b488","Type":"ContainerDied","Data":"1a8eafd93d801a2c06e6d6a47811dfa8098f32e07e0d2dd98a865c93b42fcb0c"} Nov 02 22:26:25.449771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.449717 4309 scope.go:122] "RemoveContainer" containerID="71da3f4205c4f8042aee8f59609c0f7c2e05ba77b9824dd984ca264a60507c47" Nov 02 22:26:25.498942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.498903 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s"] Nov 02 22:26:25.506350 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.505952 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5eba37-794894d995-2xx5s"] Nov 02 22:26:25.506756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.506649 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused" Nov 02 22:26:25.767081 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.767048 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:25.771303 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.771221 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:25.776587 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.776568 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:26:25.795372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.795353 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:26:25.868449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.868414 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-kube-api-access-m2z8w\" (UniqueName: \"kubernetes.io/projected/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-kube-api-access-m2z8w\") pod \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " Nov 02 22:26:25.868647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.868498 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tmp-dir\") pod \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " Nov 02 22:26:25.868647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.868561 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-dshm\") pod \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " Nov 02 22:26:25.868647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.868626 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-xks-ca\") pod \"896a2b60-040c-44f1-b84c-d6fac267b653\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " Nov 02 22:26:25.868868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.868660 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-openshift-ca\") pod \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " Nov 02 22:26:25.868868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.868722 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896a2b60-040c-44f1-b84c-d6fac267b653-kube-api-access-fhll2\" (UniqueName: \"kubernetes.io/projected/896a2b60-040c-44f1-b84c-d6fac267b653-kube-api-access-fhll2\") pod \"896a2b60-040c-44f1-b84c-d6fac267b653\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " Nov 02 22:26:25.868868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.868771 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-openshift-ca\") pod \"896a2b60-040c-44f1-b84c-d6fac267b653\" (UID: \"896a2b60-040c-44f1-b84c-d6fac267b653\") " Nov 02 22:26:25.868868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.868808 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-model-cache\") pod \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " Nov 02 22:26:25.868868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.868852 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-home\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-home\") pod \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " Nov 02 22:26:25.869140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869103 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-home" pod "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" (UID: "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:25.869196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869176 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-openshift-ca" pod "92a04793-1d76-4d00-9f00-7d7c48d6a3ac" (UID: "92a04793-1d76-4d00-9f00-7d7c48d6a3ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:25.869250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869231 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tls-certs\") pod \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " Nov 02 22:26:25.869309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869282 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tmp-dir\") pod \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " Nov 02 22:26:25.869365 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869329 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-kube-api-access-99qhb\" (UniqueName: \"kubernetes.io/projected/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-kube-api-access-99qhb\") pod \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " Nov 02 22:26:25.869419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869363 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-dshm\") pod \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\" (UID: \"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a\") " Nov 02 22:26:25.869419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869396 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-kube-api-access-t7mjw\" (UniqueName: \"kubernetes.io/projected/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-kube-api-access-t7mjw\") pod \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " Nov 02 22:26:25.869527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869430 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-xks-ca\") pod \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\" (UID: \"92a04793-1d76-4d00-9f00-7d7c48d6a3ac\") " Nov 02 22:26:25.869527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869461 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-model-cache\") pod \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " Nov 02 22:26:25.869786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869655 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tls-certs\") pod \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " Nov 02 22:26:25.869786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.869762 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-home\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-home\") pod \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\" (UID: \"bd330d8c-bd62-4cb4-a33f-b3604f55bc47\") " Nov 02 22:26:25.870436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.870408 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-openshift-ca" pod "896a2b60-040c-44f1-b84c-d6fac267b653" (UID: "896a2b60-040c-44f1-b84c-d6fac267b653"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:25.870855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.870705 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-model-cache" pod "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" (UID: "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:25.871247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.871172 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-model-cache" pod "bd330d8c-bd62-4cb4-a33f-b3604f55bc47" (UID: "bd330d8c-bd62-4cb4-a33f-b3604f55bc47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:25.871397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.871376 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tmp-dir" pod "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" (UID: "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:25.872146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.872118 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896a2b60-040c-44f1-b84c-d6fac267b653-kube-api-access-fhll2" pod "896a2b60-040c-44f1-b84c-d6fac267b653" (UID: "896a2b60-040c-44f1-b84c-d6fac267b653"). InnerVolumeSpecName "kube-api-access-fhll2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:25.874129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.874092 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-home" pod "bd330d8c-bd62-4cb4-a33f-b3604f55bc47" (UID: "bd330d8c-bd62-4cb4-a33f-b3604f55bc47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:25.874235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.874211 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-xks-ca" pod "896a2b60-040c-44f1-b84c-d6fac267b653" (UID: "896a2b60-040c-44f1-b84c-d6fac267b653"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:25.874327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.874306 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tmp-dir" pod "bd330d8c-bd62-4cb4-a33f-b3604f55bc47" (UID: "bd330d8c-bd62-4cb4-a33f-b3604f55bc47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:25.874456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.874379 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.874456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.874401 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.874622 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.874486 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-kube-api-access-t7mjw" pod "92a04793-1d76-4d00-9f00-7d7c48d6a3ac" (UID: "92a04793-1d76-4d00-9f00-7d7c48d6a3ac"). InnerVolumeSpecName "kube-api-access-t7mjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:25.874742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.874655 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tls-certs" pod "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" (UID: "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:25.876368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.876342 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-kube-api-access-99qhb" pod "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" (UID: "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a"). InnerVolumeSpecName "kube-api-access-99qhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:25.876468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.876405 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-dshm" pod "bd330d8c-bd62-4cb4-a33f-b3604f55bc47" (UID: "bd330d8c-bd62-4cb4-a33f-b3604f55bc47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:25.877026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.877001 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-xks-ca" pod "92a04793-1d76-4d00-9f00-7d7c48d6a3ac" (UID: "92a04793-1d76-4d00-9f00-7d7c48d6a3ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:25.877139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.877070 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-kube-api-access-m2z8w" pod "bd330d8c-bd62-4cb4-a33f-b3604f55bc47" (UID: "bd330d8c-bd62-4cb4-a33f-b3604f55bc47"). InnerVolumeSpecName "kube-api-access-m2z8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:25.877314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.877286 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tls-certs" pod "bd330d8c-bd62-4cb4-a33f-b3604f55bc47" (UID: "bd330d8c-bd62-4cb4-a33f-b3604f55bc47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:25.878218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.878199 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-dshm" pod "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" (UID: "d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:25.975347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975270 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975301 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975312 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2z8w\" (UniqueName: \"kubernetes.io/projected/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-kube-api-access-m2z8w\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975322 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975330 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975338 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975347 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhll2\" (UniqueName: \"kubernetes.io/projected/896a2b60-040c-44f1-b84c-d6fac267b653-kube-api-access-fhll2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975355 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896a2b60-040c-44f1-b84c-d6fac267b653-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975364 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975372 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975380 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975388 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99qhb\" (UniqueName: \"kubernetes.io/projected/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-kube-api-access-99qhb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975408 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975416 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7mjw\" (UniqueName: \"kubernetes.io/projected/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-kube-api-access-t7mjw\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975425 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92a04793-1d76-4d00-9f00-7d7c48d6a3ac-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:25.975706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:25.975432 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd330d8c-bd62-4cb4-a33f-b3604f55bc47-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:26.133597 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.133560 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" podUID="7315bae5-39b1-4241-bba0-303c9a5ad5e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:26.189333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.189303 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db96d89-7a42-4efb-aad5-f2d19119b488" path="/var/lib/kubelet/pods/7db96d89-7a42-4efb-aad5-f2d19119b488/volumes" Nov 02 22:26:26.459007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.458975 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" Nov 02 22:26:26.459456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.458981 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz" event={"ID":"92a04793-1d76-4d00-9f00-7d7c48d6a3ac","Type":"ContainerDied","Data":"5946c751dfaa551e3a7f1b9deee1449289692aa14bccc14d2832885c4e0439ec"} Nov 02 22:26:26.459456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.459124 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5946c751dfaa551e3a7f1b9deee1449289692aa14bccc14d2832885c4e0439ec" Nov 02 22:26:26.460641 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.460620 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" Nov 02 22:26:26.460775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.460653 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf" event={"ID":"896a2b60-040c-44f1-b84c-d6fac267b653","Type":"ContainerDied","Data":"a01a23354e94ca452abe2caf87e9d75ec45b3a3d365aee5e920e1b26a96cfd45"} Nov 02 22:26:26.460775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.460720 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a01a23354e94ca452abe2caf87e9d75ec45b3a3d365aee5e920e1b26a96cfd45" Nov 02 22:26:26.462715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.462657 4309 generic.go:356] "Generic (PLEG): container finished" podID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerID="9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c" exitCode=0 Nov 02 22:26:26.462807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.462713 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" event={"ID":"bd330d8c-bd62-4cb4-a33f-b3604f55bc47","Type":"ContainerDied","Data":"9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c"} Nov 02 22:26:26.462807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.462730 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" Nov 02 22:26:26.462807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.462756 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb" event={"ID":"bd330d8c-bd62-4cb4-a33f-b3604f55bc47","Type":"ContainerDied","Data":"78bcd0b13988d207d13af56e133e49bf1a731a17973e46bb26614b0f264ffb51"} Nov 02 22:26:26.462807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.462778 4309 scope.go:122] "RemoveContainer" containerID="9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c" Nov 02 22:26:26.466093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.466067 4309 generic.go:356] "Generic (PLEG): container finished" podID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerID="0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba" exitCode=0 Nov 02 22:26:26.466198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.466113 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" event={"ID":"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a","Type":"ContainerDied","Data":"0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba"} Nov 02 22:26:26.466198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.466135 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" event={"ID":"d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a","Type":"ContainerDied","Data":"a2c4b4e76bbb12a54db0d5e7ff4e7c124f770bb9c27a5f1f9576157b4e10c275"} Nov 02 22:26:26.466198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.466173 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l" Nov 02 22:26:26.475201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.475174 4309 scope.go:122] "RemoveContainer" containerID="9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c" Nov 02 22:26:26.475467 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:26.475447 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c\": container with ID starting with 9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c not found: ID does not exist" containerID="9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c" Nov 02 22:26:26.475528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.475474 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c"} err="failed to get container status \"9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c\": rpc error: code = NotFound desc = could not find container \"9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c\": container with ID starting with 9263b4fad2a7b788888ce385db5e8371aeab2e12cb7ed0a78168520a1d58516c not found: ID does not exist" Nov 02 22:26:26.475528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.475492 4309 scope.go:122] "RemoveContainer" containerID="0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba" Nov 02 22:26:26.486781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.486761 4309 scope.go:122] "RemoveContainer" containerID="0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba" Nov 02 22:26:26.487045 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:26.487020 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba\": container with ID starting with 0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba not found: ID does not exist" containerID="0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba" Nov 02 22:26:26.487145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.487048 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba"} err="failed to get container status \"0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba\": rpc error: code = NotFound desc = could not find container \"0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba\": container with ID starting with 0a6f19fb920f5e37ea010633338b75862f56533f81a5609cca58e9818a9ec6ba not found: ID does not exist" Nov 02 22:26:26.487747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.487726 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb"] Nov 02 22:26:26.491399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.491375 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/simulator-w0-310bb3-kserve-fbd8b6bf6-rxrlb"] Nov 02 22:26:26.505107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.505079 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l"] Nov 02 22:26:26.508229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.508209 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-310bb3/distinct-w0-310bb3-kserve-568cfd5dd4-g7w5l"] Nov 02 22:26:26.692684 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.692630 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" podUID="90333f30-7fb2-446f-b2b4-a9c2792c5794" containerName="payload-processing" containerID="cri-o://bc999d2836b4b27856bf0c2d42870cdbce8604109b7fb1fd2747762c838786e1" gracePeriod=25 Nov 02 22:26:26.706462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:26.706430 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" podUID="08b0dad1-c312-47fd-ac10-8b4d26a32735" containerName="payload-pre-processing" containerID="cri-o://f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4" gracePeriod=25 Nov 02 22:26:28.163206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:28.163165 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" podUID="90333f30-7fb2-446f-b2b4-a9c2792c5794" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:28.163570 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:28.163168 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" podUID="08b0dad1-c312-47fd-ac10-8b4d26a32735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:28.189500 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:28.189473 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" path="/var/lib/kubelet/pods/bd330d8c-bd62-4cb4-a33f-b3604f55bc47/volumes" Nov 02 22:26:28.190012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:28.189993 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" path="/var/lib/kubelet/pods/d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a/volumes" Nov 02 22:26:29.179712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:29.179666 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" podUID="2304b2a9-889a-4265-9ff3-25ac877f2f0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:30.860054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:30.860021 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p"] Nov 02 22:26:30.861176 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:30.861144 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" containerID="cri-o://ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d" gracePeriod=30 Nov 02 22:26:30.867424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:30.865615 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c"] Nov 02 22:26:30.867424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:30.865828 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" containerID="cri-o://bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f" gracePeriod=30 Nov 02 22:26:30.941655 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:30.941616 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc"] Nov 02 22:26:30.969795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:30.969755 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5"] Nov 02 22:26:31.334697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.332201 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" containerID="cri-o://70fd2577c2985484420e6e4bbd94d17d19140befbc0b28810372bf178857c855" gracePeriod=45 Nov 02 22:26:31.360835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.360810 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq"] Nov 02 22:26:31.361037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.361016 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" podUID="0e2512e4-37be-40b8-b5c6-7a6926903034" containerName="discovery" containerID="cri-o://90249dafc76a07dc349444f9f28377aab00d2fe0f2841494e28238c97427e361" gracePeriod=30 Nov 02 22:26:31.538644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.538058 4309 generic.go:356] "Generic (PLEG): container finished" podID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerID="70fd2577c2985484420e6e4bbd94d17d19140befbc0b28810372bf178857c855" exitCode=0 Nov 02 22:26:31.538644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.538205 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" event={"ID":"d00f0e87-7086-4d10-b9a4-713cc4948990","Type":"ContainerDied","Data":"70fd2577c2985484420e6e4bbd94d17d19140befbc0b28810372bf178857c855"} Nov 02 22:26:31.540598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.540572 4309 generic.go:356] "Generic (PLEG): container finished" podID="0e2512e4-37be-40b8-b5c6-7a6926903034" containerID="90249dafc76a07dc349444f9f28377aab00d2fe0f2841494e28238c97427e361" exitCode=0 Nov 02 22:26:31.540808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.540704 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" event={"ID":"0e2512e4-37be-40b8-b5c6-7a6926903034","Type":"ContainerDied","Data":"90249dafc76a07dc349444f9f28377aab00d2fe0f2841494e28238c97427e361"} Nov 02 22:26:31.705631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.705308 4309 patch_prober.go:29] interesting pod/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:26:31.705631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.705382 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:26:31.838930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.838862 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:26:31.876833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.876651 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-home\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-home\") pod \"d00f0e87-7086-4d10-b9a4-713cc4948990\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " Nov 02 22:26:31.878447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.877271 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-model-cache\") pod \"d00f0e87-7086-4d10-b9a4-713cc4948990\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " Nov 02 22:26:31.878447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.877474 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00f0e87-7086-4d10-b9a4-713cc4948990-kube-api-access-jj7gx\" (UniqueName: \"kubernetes.io/projected/d00f0e87-7086-4d10-b9a4-713cc4948990-kube-api-access-jj7gx\") pod \"d00f0e87-7086-4d10-b9a4-713cc4948990\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " Nov 02 22:26:31.878447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.877555 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d00f0e87-7086-4d10-b9a4-713cc4948990-tls-certs\" (UniqueName: \"kubernetes.io/secret/d00f0e87-7086-4d10-b9a4-713cc4948990-tls-certs\") pod \"d00f0e87-7086-4d10-b9a4-713cc4948990\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " Nov 02 22:26:31.878447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.877655 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-dshm\") pod \"d00f0e87-7086-4d10-b9a4-713cc4948990\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " Nov 02 22:26:31.878447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.877775 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-tmp-dir\") pod \"d00f0e87-7086-4d10-b9a4-713cc4948990\" (UID: \"d00f0e87-7086-4d10-b9a4-713cc4948990\") " Nov 02 22:26:31.878874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.878851 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-tmp-dir" pod "d00f0e87-7086-4d10-b9a4-713cc4948990" (UID: "d00f0e87-7086-4d10-b9a4-713cc4948990"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:31.879183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.879166 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-home" pod "d00f0e87-7086-4d10-b9a4-713cc4948990" (UID: "d00f0e87-7086-4d10-b9a4-713cc4948990"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:31.879443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.879427 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-model-cache" pod "d00f0e87-7086-4d10-b9a4-713cc4948990" (UID: "d00f0e87-7086-4d10-b9a4-713cc4948990"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:31.916292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.891863 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-dshm" pod "d00f0e87-7086-4d10-b9a4-713cc4948990" (UID: "d00f0e87-7086-4d10-b9a4-713cc4948990"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:31.933234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.933150 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00f0e87-7086-4d10-b9a4-713cc4948990-tls-certs" pod "d00f0e87-7086-4d10-b9a4-713cc4948990" (UID: "d00f0e87-7086-4d10-b9a4-713cc4948990"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:31.933374 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.933302 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00f0e87-7086-4d10-b9a4-713cc4948990-kube-api-access-jj7gx" pod "d00f0e87-7086-4d10-b9a4-713cc4948990" (UID: "d00f0e87-7086-4d10-b9a4-713cc4948990"). InnerVolumeSpecName "kube-api-access-jj7gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:31.939395 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.938590 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:26:31.978666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.978639 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-kubeconfig\") pod \"0e2512e4-37be-40b8-b5c6-7a6926903034\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " Nov 02 22:26:31.978853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.978780 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-ca-configmap\") pod \"0e2512e4-37be-40b8-b5c6-7a6926903034\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " Nov 02 22:26:31.978853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.978820 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2512e4-37be-40b8-b5c6-7a6926903034-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2512e4-37be-40b8-b5c6-7a6926903034-local-certs\") pod \"0e2512e4-37be-40b8-b5c6-7a6926903034\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " Nov 02 22:26:31.978959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.978851 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-cacerts\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-cacerts\") pod \"0e2512e4-37be-40b8-b5c6-7a6926903034\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " Nov 02 22:26:31.979054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.979038 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-dns-cert\") pod \"0e2512e4-37be-40b8-b5c6-7a6926903034\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " Nov 02 22:26:31.979149 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.979092 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-token\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-token\") pod \"0e2512e4-37be-40b8-b5c6-7a6926903034\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " Nov 02 22:26:31.979149 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.979126 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-kube-api-access-clccs\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-kube-api-access-clccs\") pod \"0e2512e4-37be-40b8-b5c6-7a6926903034\" (UID: \"0e2512e4-37be-40b8-b5c6-7a6926903034\") " Nov 02 22:26:31.980149 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.980123 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:31.980149 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.980151 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:31.980594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.980165 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj7gx\" (UniqueName: \"kubernetes.io/projected/d00f0e87-7086-4d10-b9a4-713cc4948990-kube-api-access-jj7gx\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:31.980594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.980179 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d00f0e87-7086-4d10-b9a4-713cc4948990-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:31.980594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.980192 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:31.980594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:31.980205 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d00f0e87-7086-4d10-b9a4-713cc4948990-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:32.015696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.014346 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-dns-cert" pod "0e2512e4-37be-40b8-b5c6-7a6926903034" (UID: "0e2512e4-37be-40b8-b5c6-7a6926903034"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:32.015696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.014877 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-ca-configmap" pod "0e2512e4-37be-40b8-b5c6-7a6926903034" (UID: "0e2512e4-37be-40b8-b5c6-7a6926903034"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:32.015696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.014978 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2512e4-37be-40b8-b5c6-7a6926903034-local-certs" pod "0e2512e4-37be-40b8-b5c6-7a6926903034" (UID: "0e2512e4-37be-40b8-b5c6-7a6926903034"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:32.017992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.017959 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-cacerts" pod "0e2512e4-37be-40b8-b5c6-7a6926903034" (UID: "0e2512e4-37be-40b8-b5c6-7a6926903034"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:32.019134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.019099 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-kube-api-access-clccs" pod "0e2512e4-37be-40b8-b5c6-7a6926903034" (UID: "0e2512e4-37be-40b8-b5c6-7a6926903034"). InnerVolumeSpecName "kube-api-access-clccs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:32.019765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.019725 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-kubeconfig" pod "0e2512e4-37be-40b8-b5c6-7a6926903034" (UID: "0e2512e4-37be-40b8-b5c6-7a6926903034"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:32.020131 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.020104 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-token" pod "0e2512e4-37be-40b8-b5c6-7a6926903034" (UID: "0e2512e4-37be-40b8-b5c6-7a6926903034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:32.081432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.081397 4309 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-dns-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:32.081432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.081427 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:32.081432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.081438 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clccs\" (UniqueName: \"kubernetes.io/projected/0e2512e4-37be-40b8-b5c6-7a6926903034-kube-api-access-clccs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:32.081660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.081447 4309 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-kubeconfig\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:32.081660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.081456 4309 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0e2512e4-37be-40b8-b5c6-7a6926903034-istio-csr-ca-configmap\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:32.081660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.081465 4309 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e2512e4-37be-40b8-b5c6-7a6926903034-local-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:32.081660 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.081475 4309 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0e2512e4-37be-40b8-b5c6-7a6926903034-cacerts\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:32.580532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.564065 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" event={"ID":"d00f0e87-7086-4d10-b9a4-713cc4948990","Type":"ContainerDied","Data":"21809fb6096eb7f558c44acf15d905dfb85efe248ae77d65d01f5dd8222928f7"} Nov 02 22:26:32.580532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.564119 4309 scope.go:122] "RemoveContainer" containerID="70fd2577c2985484420e6e4bbd94d17d19140befbc0b28810372bf178857c855" Nov 02 22:26:32.580532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.564257 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" Nov 02 22:26:32.586527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.585772 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" Nov 02 22:26:32.586527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.585775 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq" event={"ID":"0e2512e4-37be-40b8-b5c6-7a6926903034","Type":"ContainerDied","Data":"20801108b4ec63cce905546aea96084e32d9f27415336ab436cf1588449836b3"} Nov 02 22:26:32.645056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.642796 4309 scope.go:122] "RemoveContainer" containerID="90249dafc76a07dc349444f9f28377aab00d2fe0f2841494e28238c97427e361" Nov 02 22:26:32.658745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.647490 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89"] Nov 02 22:26:32.670036 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.669989 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89"] Nov 02 22:26:32.724799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.724761 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq"] Nov 02 22:26:32.730872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.730814 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fm9kq"] Nov 02 22:26:32.739809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.739777 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs"] Nov 02 22:26:32.745456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.745433 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-310bb3-cm7bs"] Nov 02 22:26:32.809065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:32.808999 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2532af/distinct-w2-2532af-kserve-746c6865b4-xdw89" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 22:26:33.353641 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.353613 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:26:33.450975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.450945 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7315bae5-39b1-4241-bba0-303c9a5ad5e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7315bae5-39b1-4241-bba0-303c9a5ad5e7-plugins-config-volume\") pod \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\" (UID: \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\") " Nov 02 22:26:33.451165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.451133 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7315bae5-39b1-4241-bba0-303c9a5ad5e7-kube-api-access-pfsrb\" (UniqueName: \"kubernetes.io/projected/7315bae5-39b1-4241-bba0-303c9a5ad5e7-kube-api-access-pfsrb\") pod \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\" (UID: \"7315bae5-39b1-4241-bba0-303c9a5ad5e7\") " Nov 02 22:26:33.451331 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.451308 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7315bae5-39b1-4241-bba0-303c9a5ad5e7-plugins-config-volume" pod "7315bae5-39b1-4241-bba0-303c9a5ad5e7" (UID: "7315bae5-39b1-4241-bba0-303c9a5ad5e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:33.451594 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.451575 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7315bae5-39b1-4241-bba0-303c9a5ad5e7-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:33.453172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.453151 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7315bae5-39b1-4241-bba0-303c9a5ad5e7-kube-api-access-pfsrb" pod "7315bae5-39b1-4241-bba0-303c9a5ad5e7" (UID: "7315bae5-39b1-4241-bba0-303c9a5ad5e7"). InnerVolumeSpecName "kube-api-access-pfsrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:33.552744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.552713 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfsrb\" (UniqueName: \"kubernetes.io/projected/7315bae5-39b1-4241-bba0-303c9a5ad5e7-kube-api-access-pfsrb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:33.629767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.629689 4309 generic.go:356] "Generic (PLEG): container finished" podID="7315bae5-39b1-4241-bba0-303c9a5ad5e7" containerID="e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b" exitCode=0 Nov 02 22:26:33.629911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.629884 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" event={"ID":"7315bae5-39b1-4241-bba0-303c9a5ad5e7","Type":"ContainerDied","Data":"e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b"} Nov 02 22:26:33.629990 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.629974 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" event={"ID":"7315bae5-39b1-4241-bba0-303c9a5ad5e7","Type":"ContainerDied","Data":"d2364e719d8b292f75f0a8e9852cacd9b7561c8d937042102f863611bef32763"} Nov 02 22:26:33.630032 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.630014 4309 scope.go:122] "RemoveContainer" containerID="e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b" Nov 02 22:26:33.630177 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.630163 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph" Nov 02 22:26:33.646261 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.646241 4309 scope.go:122] "RemoveContainer" containerID="e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b" Nov 02 22:26:33.646523 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:33.646498 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b\": container with ID starting with e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b not found: ID does not exist" containerID="e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b" Nov 02 22:26:33.646586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.646535 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b"} err="failed to get container status \"e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b\": rpc error: code = NotFound desc = could not find container \"e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b\": container with ID starting with e70d4c4dbd45201de9dc6902c73840cdfb3c2fc8b3d221c84c23ee7ef562707b not found: ID does not exist" Nov 02 22:26:33.663376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.663352 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph"] Nov 02 22:26:33.669102 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.669081 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-719012-5bf586dc85-ccfph"] Nov 02 22:26:33.841475 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.841444 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52"] Nov 02 22:26:33.844773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:33.844753 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2532af-ctm52"] Nov 02 22:26:34.212747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:34.212708 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e2512e4-37be-40b8-b5c6-7a6926903034" path="/var/lib/kubelet/pods/0e2512e4-37be-40b8-b5c6-7a6926903034/volumes" Nov 02 22:26:34.213343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:34.213323 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48074a11-e38e-4b34-a9da-db3fcfe95a84" path="/var/lib/kubelet/pods/48074a11-e38e-4b34-a9da-db3fcfe95a84/volumes" Nov 02 22:26:34.213915 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:34.213899 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d48b3c-1285-4739-b672-644d7c189a64" path="/var/lib/kubelet/pods/48d48b3c-1285-4739-b672-644d7c189a64/volumes" Nov 02 22:26:34.214787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:34.214764 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7315bae5-39b1-4241-bba0-303c9a5ad5e7" path="/var/lib/kubelet/pods/7315bae5-39b1-4241-bba0-303c9a5ad5e7/volumes" Nov 02 22:26:34.215368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:34.215350 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" path="/var/lib/kubelet/pods/d00f0e87-7086-4d10-b9a4-713cc4948990/volumes" Nov 02 22:26:35.947780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:35.945735 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" podUID="42506da4-8df1-4e2c-822b-3463e867de25" containerName="payload-processing" containerID="cri-o://ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316" gracePeriod=25 Nov 02 22:26:35.972762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:35.972720 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" podUID="4e586569-339e-4d3a-a417-496d2635c13f" containerName="payload-pre-processing" containerID="cri-o://fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2" gracePeriod=25 Nov 02 22:26:36.024396 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.019009 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng"] Nov 02 22:26:36.024396 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.019193 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" containerID="cri-o://660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4" gracePeriod=30 Nov 02 22:26:36.308138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.300856 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.392138 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.409293 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istiod-ca-cert\") pod \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.409365 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-token\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-token\") pod \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.409459 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-envoy\") pod \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.409492 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-podinfo\") pod \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.409527 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-data\") pod \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.409624 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-credential-socket\") pod \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.409711 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-socket\") pod \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.409748 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-kube-api-access-8n8gb\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-kube-api-access-8n8gb\") pod \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.409805 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-certs\") pod \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\" (UID: \"66ff0346-d7be-4890-bdfe-7c7b9d398fd5\") " Nov 02 22:26:36.415782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.413002 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-credential-socket" pod "66ff0346-d7be-4890-bdfe-7c7b9d398fd5" (UID: "66ff0346-d7be-4890-bdfe-7c7b9d398fd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.417636 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.417530 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-certs" pod "66ff0346-d7be-4890-bdfe-7c7b9d398fd5" (UID: "66ff0346-d7be-4890-bdfe-7c7b9d398fd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.417927 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.417896 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istiod-ca-cert" pod "66ff0346-d7be-4890-bdfe-7c7b9d398fd5" (UID: "66ff0346-d7be-4890-bdfe-7c7b9d398fd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:36.418270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.418208 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-socket" pod "66ff0346-d7be-4890-bdfe-7c7b9d398fd5" (UID: "66ff0346-d7be-4890-bdfe-7c7b9d398fd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.418270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.418249 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-data" pod "66ff0346-d7be-4890-bdfe-7c7b9d398fd5" (UID: "66ff0346-d7be-4890-bdfe-7c7b9d398fd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.420240 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.420121 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-envoy" pod "66ff0346-d7be-4890-bdfe-7c7b9d398fd5" (UID: "66ff0346-d7be-4890-bdfe-7c7b9d398fd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.420358 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.420306 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-podinfo" pod "66ff0346-d7be-4890-bdfe-7c7b9d398fd5" (UID: "66ff0346-d7be-4890-bdfe-7c7b9d398fd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:36.421013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.420716 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-kube-api-access-8n8gb" pod "66ff0346-d7be-4890-bdfe-7c7b9d398fd5" (UID: "66ff0346-d7be-4890-bdfe-7c7b9d398fd5"). InnerVolumeSpecName "kube-api-access-8n8gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:36.426436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.426410 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:26:36.432920 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.426829 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-token" pod "66ff0346-d7be-4890-bdfe-7c7b9d398fd5" (UID: "66ff0346-d7be-4890-bdfe-7c7b9d398fd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514157 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e586569-339e-4d3a-a417-496d2635c13f-kube-api-access-mwkvv\" (UniqueName: \"kubernetes.io/projected/4e586569-339e-4d3a-a417-496d2635c13f-kube-api-access-mwkvv\") pod \"4e586569-339e-4d3a-a417-496d2635c13f\" (UID: \"4e586569-339e-4d3a-a417-496d2635c13f\") " Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514282 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e586569-339e-4d3a-a417-496d2635c13f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e586569-339e-4d3a-a417-496d2635c13f-plugins-config-volume\") pod \"4e586569-339e-4d3a-a417-496d2635c13f\" (UID: \"4e586569-339e-4d3a-a417-496d2635c13f\") " Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514389 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42506da4-8df1-4e2c-822b-3463e867de25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42506da4-8df1-4e2c-822b-3463e867de25-plugins-config-volume\") pod \"42506da4-8df1-4e2c-822b-3463e867de25\" (UID: \"42506da4-8df1-4e2c-822b-3463e867de25\") " Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514428 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42506da4-8df1-4e2c-822b-3463e867de25-kube-api-access-8gmkd\" (UniqueName: \"kubernetes.io/projected/42506da4-8df1-4e2c-822b-3463e867de25-kube-api-access-8gmkd\") pod \"42506da4-8df1-4e2c-822b-3463e867de25\" (UID: \"42506da4-8df1-4e2c-822b-3463e867de25\") " Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514886 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514897 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e586569-339e-4d3a-a417-496d2635c13f-plugins-config-volume" pod "4e586569-339e-4d3a-a417-496d2635c13f" (UID: "4e586569-339e-4d3a-a417-496d2635c13f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514911 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514927 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n8gb\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-kube-api-access-8n8gb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514943 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514955 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514970 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514983 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.514997 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.515266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.515010 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ff0346-d7be-4890-bdfe-7c7b9d398fd5-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.516514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.515699 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42506da4-8df1-4e2c-822b-3463e867de25-plugins-config-volume" pod "42506da4-8df1-4e2c-822b-3463e867de25" (UID: "42506da4-8df1-4e2c-822b-3463e867de25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:36.517461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.517438 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e586569-339e-4d3a-a417-496d2635c13f-kube-api-access-mwkvv" pod "4e586569-339e-4d3a-a417-496d2635c13f" (UID: "4e586569-339e-4d3a-a417-496d2635c13f"). InnerVolumeSpecName "kube-api-access-mwkvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:36.518496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.518474 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42506da4-8df1-4e2c-822b-3463e867de25-kube-api-access-8gmkd" pod "42506da4-8df1-4e2c-822b-3463e867de25" (UID: "42506da4-8df1-4e2c-822b-3463e867de25"). InnerVolumeSpecName "kube-api-access-8gmkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:36.632080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.626059 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwkvv\" (UniqueName: \"kubernetes.io/projected/4e586569-339e-4d3a-a417-496d2635c13f-kube-api-access-mwkvv\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.632080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.626093 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e586569-339e-4d3a-a417-496d2635c13f-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.632080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.626108 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42506da4-8df1-4e2c-822b-3463e867de25-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.632080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.626122 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gmkd\" (UniqueName: \"kubernetes.io/projected/42506da4-8df1-4e2c-822b-3463e867de25-kube-api-access-8gmkd\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.662087 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.662052 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:26:36.664319 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.664293 4309 generic.go:356] "Generic (PLEG): container finished" podID="42506da4-8df1-4e2c-822b-3463e867de25" containerID="ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316" exitCode=0 Nov 02 22:26:36.664397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.664367 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" event={"ID":"42506da4-8df1-4e2c-822b-3463e867de25","Type":"ContainerDied","Data":"ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316"} Nov 02 22:26:36.664439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.664403 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" event={"ID":"42506da4-8df1-4e2c-822b-3463e867de25","Type":"ContainerDied","Data":"504b81f6028a889e53f698d5232ef149730078583c69f7bfddd498ee24ecb10b"} Nov 02 22:26:36.664439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.664422 4309 scope.go:122] "RemoveContainer" containerID="ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316" Nov 02 22:26:36.664506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.664487 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc" Nov 02 22:26:36.670581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.670556 4309 generic.go:356] "Generic (PLEG): container finished" podID="4e586569-339e-4d3a-a417-496d2635c13f" containerID="fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2" exitCode=0 Nov 02 22:26:36.670665 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.670596 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" Nov 02 22:26:36.670742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.670708 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" event={"ID":"4e586569-339e-4d3a-a417-496d2635c13f","Type":"ContainerDied","Data":"fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2"} Nov 02 22:26:36.670795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.670750 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5" event={"ID":"4e586569-339e-4d3a-a417-496d2635c13f","Type":"ContainerDied","Data":"9366e852ccc65606479762657c39030c63d1558d2f6887f276e3724e419a8f8a"} Nov 02 22:26:36.672433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.672407 4309 generic.go:356] "Generic (PLEG): container finished" podID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerID="bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f" exitCode=0 Nov 02 22:26:36.672537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.672445 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" Nov 02 22:26:36.673022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.672580 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" event={"ID":"c35ad342-26b9-43a9-a327-02309df9f1dc","Type":"ContainerDied","Data":"bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f"} Nov 02 22:26:36.673022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.672622 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c" event={"ID":"c35ad342-26b9-43a9-a327-02309df9f1dc","Type":"ContainerDied","Data":"23fa0d28a17bf98c1ae812e597213871ca35ba9bc20a068591c31c1ce58c2608"} Nov 02 22:26:36.675481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.675457 4309 generic.go:356] "Generic (PLEG): container finished" podID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerID="ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d" exitCode=0 Nov 02 22:26:36.675590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.675517 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" event={"ID":"66ff0346-d7be-4890-bdfe-7c7b9d398fd5","Type":"ContainerDied","Data":"ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d"} Nov 02 22:26:36.675590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.675542 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" event={"ID":"66ff0346-d7be-4890-bdfe-7c7b9d398fd5","Type":"ContainerDied","Data":"04badde7c7f1c08c5091fca2914819489375c0211e5968d791a750bea8edacc4"} Nov 02 22:26:36.675590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.675588 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p" Nov 02 22:26:36.682439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.682366 4309 scope.go:122] "RemoveContainer" containerID="ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316" Nov 02 22:26:36.683157 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:36.683133 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316\": container with ID starting with ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316 not found: ID does not exist" containerID="ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316" Nov 02 22:26:36.683922 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.683893 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316"} err="failed to get container status \"ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316\": rpc error: code = NotFound desc = could not find container \"ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316\": container with ID starting with ba343bfc505dcc28078206a0376b7ebd407f7fff6b254b583d6278b145cc9316 not found: ID does not exist" Nov 02 22:26:36.684063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.684048 4309 scope.go:122] "RemoveContainer" containerID="fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2" Nov 02 22:26:36.753888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.753863 4309 scope.go:122] "RemoveContainer" containerID="fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2" Nov 02 22:26:36.755635 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:36.755437 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2\": container with ID starting with fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2 not found: ID does not exist" containerID="fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2" Nov 02 22:26:36.755635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.755472 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2"} err="failed to get container status \"fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2\": rpc error: code = NotFound desc = could not find container \"fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2\": container with ID starting with fe23cbdefac9aec728b5a2246e401fe1cc494d9040b55bd787bc1623037b0dd2 not found: ID does not exist" Nov 02 22:26:36.755635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.755494 4309 scope.go:122] "RemoveContainer" containerID="bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f" Nov 02 22:26:36.755635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.755562 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc"] Nov 02 22:26:36.760384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.760363 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a3713a-58d8d9685-ltrbc"] Nov 02 22:26:36.765629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.765582 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p"] Nov 02 22:26:36.772693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.770138 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-65x6p"] Nov 02 22:26:36.772693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.772293 4309 scope.go:122] "RemoveContainer" containerID="bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f" Nov 02 22:26:36.772693 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:36.772615 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f\": container with ID starting with bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f not found: ID does not exist" containerID="bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f" Nov 02 22:26:36.772693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.772649 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f"} err="failed to get container status \"bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f\": rpc error: code = NotFound desc = could not find container \"bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f\": container with ID starting with bb75dbd3dc9c6cdd91736f6dbcf76a084375989e6991a9a2d6fd626f2146434f not found: ID does not exist" Nov 02 22:26:36.772693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.772688 4309 scope.go:122] "RemoveContainer" containerID="ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d" Nov 02 22:26:36.777365 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.777340 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5"] Nov 02 22:26:36.785240 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.785220 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-a3713a-8d86ff4f5-66lp5"] Nov 02 22:26:36.818453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.818431 4309 scope.go:122] "RemoveContainer" containerID="ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035" Nov 02 22:26:36.842461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.840295 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-kube-api-access-ww5fq\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-kube-api-access-ww5fq\") pod \"c35ad342-26b9-43a9-a327-02309df9f1dc\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " Nov 02 22:26:36.842461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.840366 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-envoy\") pod \"c35ad342-26b9-43a9-a327-02309df9f1dc\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " Nov 02 22:26:36.842461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.840411 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-token\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-token\") pod \"c35ad342-26b9-43a9-a327-02309df9f1dc\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " Nov 02 22:26:36.842461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.840493 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-credential-socket\") pod \"c35ad342-26b9-43a9-a327-02309df9f1dc\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " Nov 02 22:26:36.842461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.840546 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-data\") pod \"c35ad342-26b9-43a9-a327-02309df9f1dc\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " Nov 02 22:26:36.842461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.840658 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-certs\") pod \"c35ad342-26b9-43a9-a327-02309df9f1dc\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " Nov 02 22:26:36.842461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.840737 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-socket\") pod \"c35ad342-26b9-43a9-a327-02309df9f1dc\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " Nov 02 22:26:36.842461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.840788 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35ad342-26b9-43a9-a327-02309df9f1dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35ad342-26b9-43a9-a327-02309df9f1dc-istiod-ca-cert\") pod \"c35ad342-26b9-43a9-a327-02309df9f1dc\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " Nov 02 22:26:36.842461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.840827 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-podinfo\") pod \"c35ad342-26b9-43a9-a327-02309df9f1dc\" (UID: \"c35ad342-26b9-43a9-a327-02309df9f1dc\") " Nov 02 22:26:36.843947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.843557 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-credential-socket" pod "c35ad342-26b9-43a9-a327-02309df9f1dc" (UID: "c35ad342-26b9-43a9-a327-02309df9f1dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.847223 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.845330 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-socket" pod "c35ad342-26b9-43a9-a327-02309df9f1dc" (UID: "c35ad342-26b9-43a9-a327-02309df9f1dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.847223 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.847181 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35ad342-26b9-43a9-a327-02309df9f1dc-istiod-ca-cert" pod "c35ad342-26b9-43a9-a327-02309df9f1dc" (UID: "c35ad342-26b9-43a9-a327-02309df9f1dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:36.847724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.847700 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-data" pod "c35ad342-26b9-43a9-a327-02309df9f1dc" (UID: "c35ad342-26b9-43a9-a327-02309df9f1dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.848054 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.848033 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-certs" pod "c35ad342-26b9-43a9-a327-02309df9f1dc" (UID: "c35ad342-26b9-43a9-a327-02309df9f1dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.851078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.851039 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-token" pod "c35ad342-26b9-43a9-a327-02309df9f1dc" (UID: "c35ad342-26b9-43a9-a327-02309df9f1dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:36.852650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.852324 4309 scope.go:122] "RemoveContainer" containerID="ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d" Nov 02 22:26:36.852650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.852615 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-podinfo" pod "c35ad342-26b9-43a9-a327-02309df9f1dc" (UID: "c35ad342-26b9-43a9-a327-02309df9f1dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:36.853172 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:36.853031 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d\": container with ID starting with ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d not found: ID does not exist" containerID="ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d" Nov 02 22:26:36.853172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.853070 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d"} err="failed to get container status \"ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d\": rpc error: code = NotFound desc = could not find container \"ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d\": container with ID starting with ec659e02301c3ccfcb3e9cdb5cff9f2872f05a2d6b436edcfed02c72dd54eb7d not found: ID does not exist" Nov 02 22:26:36.853172 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.853093 4309 scope.go:122] "RemoveContainer" containerID="ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035" Nov 02 22:26:36.853564 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:36.853463 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035\": container with ID starting with ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035 not found: ID does not exist" containerID="ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035" Nov 02 22:26:36.853564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.853493 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035"} err="failed to get container status \"ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035\": rpc error: code = NotFound desc = could not find container \"ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035\": container with ID starting with ae48c3406c4ffb99000b595960333547ed5f027ad75632034b58d7bd86b6d035 not found: ID does not exist" Nov 02 22:26:36.853799 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.853652 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-envoy" pod "c35ad342-26b9-43a9-a327-02309df9f1dc" (UID: "c35ad342-26b9-43a9-a327-02309df9f1dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:36.854098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.854071 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-kube-api-access-ww5fq" pod "c35ad342-26b9-43a9-a327-02309df9f1dc" (UID: "c35ad342-26b9-43a9-a327-02309df9f1dc"). InnerVolumeSpecName "kube-api-access-ww5fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:36.942344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.942300 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.942344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.942339 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.942344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.942352 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.942606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.942365 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.942606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.942378 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c35ad342-26b9-43a9-a327-02309df9f1dc-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.942606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.942390 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.942606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.942401 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5fq\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-kube-api-access-ww5fq\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.942606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.942413 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.942606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.942424 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35ad342-26b9-43a9-a327-02309df9f1dc-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:36.999325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:36.999280 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c"] Nov 02 22:26:37.003314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:37.003284 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-719012-openshift-default-88dc6759f-82p8c"] Nov 02 22:26:38.167137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:38.166935 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" podUID="90333f30-7fb2-446f-b2b4-a9c2792c5794" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:38.167137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:38.166935 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" podUID="08b0dad1-c312-47fd-ac10-8b4d26a32735" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:38.190880 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:38.190843 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42506da4-8df1-4e2c-822b-3463e867de25" path="/var/lib/kubelet/pods/42506da4-8df1-4e2c-822b-3463e867de25/volumes" Nov 02 22:26:38.191513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:38.191490 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e586569-339e-4d3a-a417-496d2635c13f" path="/var/lib/kubelet/pods/4e586569-339e-4d3a-a417-496d2635c13f/volumes" Nov 02 22:26:38.192192 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:38.192171 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" path="/var/lib/kubelet/pods/66ff0346-d7be-4890-bdfe-7c7b9d398fd5/volumes" Nov 02 22:26:38.193347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:38.193324 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" path="/var/lib/kubelet/pods/c35ad342-26b9-43a9-a327-02309df9f1dc/volumes" Nov 02 22:26:39.179693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:39.179641 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" podUID="2304b2a9-889a-4265-9ff3-25ac877f2f0d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:26:40.718465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:40.718427 4309 generic.go:356] "Generic (PLEG): container finished" podID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerID="facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5" exitCode=0 Nov 02 22:26:40.719004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:40.718492 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" event={"ID":"04f9e420-c9ca-4a32-bf79-6d6d62097e9a","Type":"ContainerDied","Data":"facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5"} Nov 02 22:26:40.719004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:40.718930 4309 scope.go:122] "RemoveContainer" containerID="facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5" Nov 02 22:26:40.720460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:40.720430 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:26:40.720582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:40.720517 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:26:40.720582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:40.720564 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:26:41.004895 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.004866 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:26:41.104819 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.104780 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2304b2a9-889a-4265-9ff3-25ac877f2f0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2304b2a9-889a-4265-9ff3-25ac877f2f0d-plugins-config-volume\") pod \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\" (UID: \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\") " Nov 02 22:26:41.105026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.104877 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2304b2a9-889a-4265-9ff3-25ac877f2f0d-kube-api-access-ljfl8\" (UniqueName: \"kubernetes.io/projected/2304b2a9-889a-4265-9ff3-25ac877f2f0d-kube-api-access-ljfl8\") pod \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\" (UID: \"2304b2a9-889a-4265-9ff3-25ac877f2f0d\") " Nov 02 22:26:41.106125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.106064 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2304b2a9-889a-4265-9ff3-25ac877f2f0d-plugins-config-volume" pod "2304b2a9-889a-4265-9ff3-25ac877f2f0d" (UID: "2304b2a9-889a-4265-9ff3-25ac877f2f0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:41.107864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.107830 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2304b2a9-889a-4265-9ff3-25ac877f2f0d-kube-api-access-ljfl8" pod "2304b2a9-889a-4265-9ff3-25ac877f2f0d" (UID: "2304b2a9-889a-4265-9ff3-25ac877f2f0d"). InnerVolumeSpecName "kube-api-access-ljfl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:41.206543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.206497 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2304b2a9-889a-4265-9ff3-25ac877f2f0d-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.206543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.206540 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljfl8\" (UniqueName: \"kubernetes.io/projected/2304b2a9-889a-4265-9ff3-25ac877f2f0d-kube-api-access-ljfl8\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.321241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.317738 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm"] Nov 02 22:26:41.321241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.317971 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" containerID="cri-o://a0ac5e26ec29ea4c810fc2cdc05cc281bd3981da2ddc9e95fbaccf6a662b5c45" gracePeriod=30 Nov 02 22:26:41.321241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.321099 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7"] Nov 02 22:26:41.321556 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.321293 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" containerID="cri-o://ae59824fadb39d0f9e385e87350553b244a2c3c4165461189ebd39e8942e3bb8" gracePeriod=30 Nov 02 22:26:41.352560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.351101 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.520876 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-socket\") pod \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.520940 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-certs\") pod \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.520982 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-kube-api-access-tpxwq\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-kube-api-access-tpxwq\") pod \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.521014 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-data\") pod \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.521053 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-podinfo\") pod \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.521098 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-credential-socket\") pod \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.521173 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-token\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-token\") pod \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.521194 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istiod-ca-cert\") pod \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.521244 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-envoy\") pod \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\" (UID: \"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2\") " Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.521926 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-certs" pod "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" (UID: "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.522173 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-socket" pod "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" (UID: "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.522327 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-credential-socket" pod "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" (UID: "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:41.523722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.523019 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istiod-ca-cert" pod "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" (UID: "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:41.528706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.525881 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-envoy" pod "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" (UID: "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:41.532705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.532558 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-data" pod "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" (UID: "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:41.553111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.538782 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-podinfo" pod "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" (UID: "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:41.553111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.547948 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-token" pod "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" (UID: "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:41.553111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.548065 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-kube-api-access-tpxwq" pod "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" (UID: "ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2"). InnerVolumeSpecName "kube-api-access-tpxwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:41.612724 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:41.608614 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:26:41.622162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.622131 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.622162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.622160 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.622162 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.622170 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.622405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.622179 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.622405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.622192 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.622405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.622206 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxwq\" (UniqueName: \"kubernetes.io/projected/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-kube-api-access-tpxwq\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.622405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.622215 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.622405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.622224 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.622405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.622233 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:41.730351 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.730312 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" event={"ID":"04f9e420-c9ca-4a32-bf79-6d6d62097e9a","Type":"ContainerStarted","Data":"ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5"} Nov 02 22:26:41.732115 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.732085 4309 generic.go:356] "Generic (PLEG): container finished" podID="2304b2a9-889a-4265-9ff3-25ac877f2f0d" containerID="7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7" exitCode=0 Nov 02 22:26:41.732280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.732123 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" Nov 02 22:26:41.732280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.732161 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" event={"ID":"2304b2a9-889a-4265-9ff3-25ac877f2f0d","Type":"ContainerDied","Data":"7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7"} Nov 02 22:26:41.732280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.732201 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4" event={"ID":"2304b2a9-889a-4265-9ff3-25ac877f2f0d","Type":"ContainerDied","Data":"70b0a68596dd623cb6455d6c9327a2c14994181997863fb272cbf1eea10ed848"} Nov 02 22:26:41.732280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.732221 4309 scope.go:122] "RemoveContainer" containerID="7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7" Nov 02 22:26:41.734496 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.734476 4309 generic.go:356] "Generic (PLEG): container finished" podID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerID="660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4" exitCode=0 Nov 02 22:26:41.734733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.734667 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" event={"ID":"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2","Type":"ContainerDied","Data":"660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4"} Nov 02 22:26:41.734882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.734867 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" event={"ID":"ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2","Type":"ContainerDied","Data":"fa2b00d7ca1824538a0f74cededa3578e3bd736329f01f0844325b653d66c7f0"} Nov 02 22:26:41.735041 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.735026 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng" Nov 02 22:26:41.746523 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.746506 4309 scope.go:122] "RemoveContainer" containerID="7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7" Nov 02 22:26:41.746966 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:41.746853 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7\": container with ID starting with 7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7 not found: ID does not exist" containerID="7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7" Nov 02 22:26:41.746966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.746881 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7"} err="failed to get container status \"7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7\": rpc error: code = NotFound desc = could not find container \"7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7\": container with ID starting with 7fa7b7f03fb8f103b066dd2b513c84d6022b1feca42bd506493b124e8ac1d8d7 not found: ID does not exist" Nov 02 22:26:41.746966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.746898 4309 scope.go:122] "RemoveContainer" containerID="660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4" Nov 02 22:26:41.757606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.757585 4309 scope.go:122] "RemoveContainer" containerID="660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4" Nov 02 22:26:41.757991 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:41.757967 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4\": container with ID starting with 660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4 not found: ID does not exist" containerID="660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4" Nov 02 22:26:41.758075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.757998 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4"} err="failed to get container status \"660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4\": rpc error: code = NotFound desc = could not find container \"660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4\": container with ID starting with 660a36b1eee5972337c3204aef28ac8f72686af2251b3e766b19a4b065701db4 not found: ID does not exist" Nov 02 22:26:41.834267 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.834234 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4"] Nov 02 22:26:41.840290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.840257 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-310bb3-759db68847-l92x4"] Nov 02 22:26:41.849187 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.849161 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng"] Nov 02 22:26:41.852046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:41.852014 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r5hng"] Nov 02 22:26:42.210113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:42.209696 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2304b2a9-889a-4265-9ff3-25ac877f2f0d" path="/var/lib/kubelet/pods/2304b2a9-889a-4265-9ff3-25ac877f2f0d/volumes" Nov 02 22:26:42.218270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:42.210370 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" path="/var/lib/kubelet/pods/ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2/volumes" Nov 02 22:26:42.339436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:42.339089 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj"] Nov 02 22:26:42.339436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:42.339334 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" containerID="cri-o://f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778" gracePeriod=30 Nov 02 22:26:42.504758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:42.498070 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:26:42.504758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:42.499852 4309 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 22:26:42.504758 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:42.499905 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 22:26:42.929028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:42.928992 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f"] Nov 02 22:26:43.508777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.504116 4309 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 22:26:43.508777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.504179 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 22:26:43.552230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.552199 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:26:43.616762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.606378 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr"] Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626244 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626301 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626323 4309 cpu_manager.go:414] "containerMap: removing container" podUID="0e2512e4-37be-40b8-b5c6-7a6926903034" containerName="discovery" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626331 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2512e4-37be-40b8-b5c6-7a6926903034" containerName="discovery" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626354 4309 cpu_manager.go:414] "containerMap: removing container" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626390 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626400 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626408 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626428 4309 cpu_manager.go:414] "containerMap: removing container" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626435 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626452 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7db96d89-7a42-4efb-aad5-f2d19119b488" containerName="payload-pre-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626459 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db96d89-7a42-4efb-aad5-f2d19119b488" containerName="payload-pre-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626474 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7315bae5-39b1-4241-bba0-303c9a5ad5e7" containerName="payload-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626481 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7315bae5-39b1-4241-bba0-303c9a5ad5e7" containerName="payload-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626495 4309 cpu_manager.go:414] "containerMap: removing container" podUID="2304b2a9-889a-4265-9ff3-25ac877f2f0d" containerName="payload-pre-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626502 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2304b2a9-889a-4265-9ff3-25ac877f2f0d" containerName="payload-pre-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626514 4309 cpu_manager.go:414] "containerMap: removing container" podUID="4e586569-339e-4d3a-a417-496d2635c13f" containerName="payload-pre-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626521 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e586569-339e-4d3a-a417-496d2635c13f" containerName="payload-pre-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626533 4309 cpu_manager.go:414] "containerMap: removing container" podUID="896a2b60-040c-44f1-b84c-d6fac267b653" containerName="revoke-keys" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626540 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896a2b60-040c-44f1-b84c-d6fac267b653" containerName="revoke-keys" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626549 4309 cpu_manager.go:414] "containerMap: removing container" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626556 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626572 4309 cpu_manager.go:414] "containerMap: removing container" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626579 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626589 4309 cpu_manager.go:414] "containerMap: removing container" podUID="92a04793-1d76-4d00-9f00-7d7c48d6a3ac" containerName="revoke-keys" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626596 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a04793-1d76-4d00-9f00-7d7c48d6a3ac" containerName="revoke-keys" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626608 4309 cpu_manager.go:414] "containerMap: removing container" podUID="42506da4-8df1-4e2c-822b-3463e867de25" containerName="payload-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626615 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42506da4-8df1-4e2c-822b-3463e867de25" containerName="payload-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626637 4309 cpu_manager.go:414] "containerMap: removing container" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626645 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626785 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd330d8c-bd62-4cb4-a33f-b3604f55bc47" containerName="main" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626800 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db96d89-7a42-4efb-aad5-f2d19119b488" containerName="payload-pre-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626810 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7315bae5-39b1-4241-bba0-303c9a5ad5e7" containerName="payload-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626821 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="2304b2a9-889a-4265-9ff3-25ac877f2f0d" containerName="payload-pre-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626831 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e586569-339e-4d3a-a417-496d2635c13f" containerName="payload-pre-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626839 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="896a2b60-040c-44f1-b84c-d6fac267b653" containerName="revoke-keys" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626850 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35ad342-26b9-43a9-a327-02309df9f1dc" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626861 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4b06dc-58c0-49b7-af85-ef49b1fc5ff2" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626870 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a04793-1d76-4d00-9f00-7d7c48d6a3ac" containerName="revoke-keys" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626877 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="42506da4-8df1-4e2c-822b-3463e867de25" containerName="payload-processing" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626888 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626899 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00f0e87-7086-4d10-b9a4-713cc4948990" containerName="main" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626907 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2512e4-37be-40b8-b5c6-7a6926903034" containerName="discovery" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626919 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ff0346-d7be-4890-bdfe-7c7b9d398fd5" containerName="istio-proxy" Nov 02 22:26:43.629844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.626928 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d18a5ea4-60d8-43eb-a7c2-b43f946d0a9a" containerName="main" Nov 02 22:26:43.659661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.659491 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr"] Nov 02 22:26:43.659661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.659586 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.663310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.663193 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:26:43.663310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.663253 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:26:43.663310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.663271 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:43.664355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.664334 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:26:43.756402 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.756369 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerName="istio-proxy" containerID="cri-o://ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5" gracePeriod=30 Nov 02 22:26:43.760760 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.760703 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.760870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.760770 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tbzm\" (UniqueName: \"kubernetes.io/projected/bfb69aa7-7281-477d-9307-61d2bc250f02-kube-api-access-2tbzm\") pod \"maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.760870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.760827 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.862543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.862505 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.862744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.862591 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tbzm\" (UniqueName: \"kubernetes.io/projected/bfb69aa7-7281-477d-9307-61d2bc250f02-kube-api-access-2tbzm\") pod \"maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.862744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.862665 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.863255 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.863232 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.864845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.864824 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.872245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.872225 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tbzm\" (UniqueName: \"kubernetes.io/projected/bfb69aa7-7281-477d-9307-61d2bc250f02-kube-api-access-2tbzm\") pod \"maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:43.974773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:43.974740 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:44.309152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:44.308900 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr"] Nov 02 22:26:44.766449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:44.766410 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" event={"ID":"bfb69aa7-7281-477d-9307-61d2bc250f02","Type":"ContainerStarted","Data":"96cca41669ac0d6d4dedb66a8ff308a5da45c5a925ef1d041478657168e1a7a4"} Nov 02 22:26:44.766617 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:44.766480 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" event={"ID":"bfb69aa7-7281-477d-9307-61d2bc250f02","Type":"ContainerStarted","Data":"5c99d006bc869e37071c6602329598d56dd316992109748f92228b65fb8e3f15"} Nov 02 22:26:44.811066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:44.811000 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" podStartSLOduration=1.8109766729999999 podStartE2EDuration="1.810976673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:26:44.794042111 +0000 UTC m=+1732.732155330" watchObservedRunningTime="2026-11-02 22:26:44.810976673 +0000 UTC m=+1732.749089885" Nov 02 22:26:45.780961 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:45.780917 4309 generic.go:356] "Generic (PLEG): container finished" podID="bfb69aa7-7281-477d-9307-61d2bc250f02" containerID="96cca41669ac0d6d4dedb66a8ff308a5da45c5a925ef1d041478657168e1a7a4" exitCode=0 Nov 02 22:26:45.781360 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:45.780998 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" event={"ID":"bfb69aa7-7281-477d-9307-61d2bc250f02","Type":"ContainerDied","Data":"96cca41669ac0d6d4dedb66a8ff308a5da45c5a925ef1d041478657168e1a7a4"} Nov 02 22:26:46.457589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.457559 4309 patch_prober.go:29] interesting pod/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Nov 02 22:26:46.457772 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.457636 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Nov 02 22:26:46.829657 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.829622 4309 generic.go:356] "Generic (PLEG): container finished" podID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerID="a0ac5e26ec29ea4c810fc2cdc05cc281bd3981da2ddc9e95fbaccf6a662b5c45" exitCode=0 Nov 02 22:26:46.830109 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.829995 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" event={"ID":"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0","Type":"ContainerDied","Data":"a0ac5e26ec29ea4c810fc2cdc05cc281bd3981da2ddc9e95fbaccf6a662b5c45"} Nov 02 22:26:46.830109 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.830051 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" event={"ID":"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0","Type":"ContainerDied","Data":"bc2f8aa122c6aae3713e9811697326a5545cc32fa22f7fce38c96d003144bfc7"} Nov 02 22:26:46.830109 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.830075 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc2f8aa122c6aae3713e9811697326a5545cc32fa22f7fce38c96d003144bfc7" Nov 02 22:26:46.832632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.832602 4309 generic.go:356] "Generic (PLEG): container finished" podID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerID="ae59824fadb39d0f9e385e87350553b244a2c3c4165461189ebd39e8942e3bb8" exitCode=0 Nov 02 22:26:46.832764 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.832740 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" event={"ID":"b4e82e0c-d7b3-4881-918d-591dc346c9e2","Type":"ContainerDied","Data":"ae59824fadb39d0f9e385e87350553b244a2c3c4165461189ebd39e8942e3bb8"} Nov 02 22:26:46.836711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.836670 4309 generic.go:356] "Generic (PLEG): container finished" podID="90333f30-7fb2-446f-b2b4-a9c2792c5794" containerID="bc999d2836b4b27856bf0c2d42870cdbce8604109b7fb1fd2747762c838786e1" exitCode=0 Nov 02 22:26:46.836944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.836925 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" event={"ID":"90333f30-7fb2-446f-b2b4-a9c2792c5794","Type":"ContainerDied","Data":"bc999d2836b4b27856bf0c2d42870cdbce8604109b7fb1fd2747762c838786e1"} Nov 02 22:26:46.865171 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.864819 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:26:46.871837 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.871608 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930102 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-data\") pod \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930159 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-data\") pod \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930194 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-credential-socket\") pod \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930227 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istiod-ca-cert\") pod \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930266 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-kube-api-access-cjhgw\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-kube-api-access-cjhgw\") pod \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930313 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-socket\") pod \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930348 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-envoy\") pod \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930388 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-certs\") pod \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930417 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-envoy\") pod \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930458 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-podinfo\") pod \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930509 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-kube-api-access-gvmc8\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-kube-api-access-gvmc8\") pod \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930548 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-credential-socket\") pod \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930581 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-podinfo\") pod \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930648 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-token\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-token\") pod \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930697 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-socket\") pod \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930810 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-certs\") pod \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930839 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-token\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-token\") pod \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\" (UID: \"b4e82e0c-d7b3-4881-918d-591dc346c9e2\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.930881 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istiod-ca-cert\") pod \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\" (UID: \"d82d2ee2-8555-429a-968b-eb6bb6a1f2c0\") " Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.931883 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istiod-ca-cert" pod "b4e82e0c-d7b3-4881-918d-591dc346c9e2" (UID: "b4e82e0c-d7b3-4881-918d-591dc346c9e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.932397 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-data" pod "b4e82e0c-d7b3-4881-918d-591dc346c9e2" (UID: "b4e82e0c-d7b3-4881-918d-591dc346c9e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.932617 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-socket" pod "b4e82e0c-d7b3-4881-918d-591dc346c9e2" (UID: "b4e82e0c-d7b3-4881-918d-591dc346c9e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.934465 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-credential-socket" pod "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" (UID: "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.935127 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-certs" pod "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" (UID: "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.939872 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-certs" pod "b4e82e0c-d7b3-4881-918d-591dc346c9e2" (UID: "b4e82e0c-d7b3-4881-918d-591dc346c9e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.939925 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-data" pod "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" (UID: "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.940432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.940120 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-credential-socket" pod "b4e82e0c-d7b3-4881-918d-591dc346c9e2" (UID: "b4e82e0c-d7b3-4881-918d-591dc346c9e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.946022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.942763 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-socket" pod "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" (UID: "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.946022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.945407 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istiod-ca-cert" pod "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" (UID: "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:46.949349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.946856 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-kube-api-access-gvmc8" pod "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" (UID: "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0"). InnerVolumeSpecName "kube-api-access-gvmc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:46.949349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.948651 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-podinfo" pod "b4e82e0c-d7b3-4881-918d-591dc346c9e2" (UID: "b4e82e0c-d7b3-4881-918d-591dc346c9e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:46.952697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.951296 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-token" pod "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" (UID: "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:46.958936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.953427 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-podinfo" pod "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" (UID: "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:46.958936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.953800 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-envoy" pod "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" (UID: "d82d2ee2-8555-429a-968b-eb6bb6a1f2c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.958936 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.957348 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-token" pod "b4e82e0c-d7b3-4881-918d-591dc346c9e2" (UID: "b4e82e0c-d7b3-4881-918d-591dc346c9e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:46.960633 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.960595 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-envoy" pod "b4e82e0c-d7b3-4881-918d-591dc346c9e2" (UID: "b4e82e0c-d7b3-4881-918d-591dc346c9e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:46.960768 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:46.960646 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-kube-api-access-cjhgw" pod "b4e82e0c-d7b3-4881-918d-591dc346c9e2" (UID: "b4e82e0c-d7b3-4881-918d-591dc346c9e2"). InnerVolumeSpecName "kube-api-access-cjhgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.036298 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.036330 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.036343 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037518 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037561 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037577 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037592 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037605 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037620 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037634 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037649 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjhgw\" (UniqueName: \"kubernetes.io/projected/b4e82e0c-d7b3-4881-918d-591dc346c9e2-kube-api-access-cjhgw\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037663 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037705 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037723 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037738 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037752 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037753 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037766 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvmc8\" (UniqueName: \"kubernetes.io/projected/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0-kube-api-access-gvmc8\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.037828 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.037781 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4e82e0c-d7b3-4881-918d-591dc346c9e2-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.130333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.129954 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:47.133913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.133843 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:26:47.139233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.139208 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb69aa7-7281-477d-9307-61d2bc250f02-kube-api-access-2tbzm\" (UniqueName: \"kubernetes.io/projected/bfb69aa7-7281-477d-9307-61d2bc250f02-kube-api-access-2tbzm\") pod \"bfb69aa7-7281-477d-9307-61d2bc250f02\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " Nov 02 22:26:47.139335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.139269 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-openshift-ca\") pod \"bfb69aa7-7281-477d-9307-61d2bc250f02\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " Nov 02 22:26:47.139378 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.139351 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-xks-ca\") pod \"bfb69aa7-7281-477d-9307-61d2bc250f02\" (UID: \"bfb69aa7-7281-477d-9307-61d2bc250f02\") " Nov 02 22:26:47.139514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.139453 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90333f30-7fb2-446f-b2b4-a9c2792c5794-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90333f30-7fb2-446f-b2b4-a9c2792c5794-plugins-config-volume\") pod \"90333f30-7fb2-446f-b2b4-a9c2792c5794\" (UID: \"90333f30-7fb2-446f-b2b4-a9c2792c5794\") " Nov 02 22:26:47.139588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.139520 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90333f30-7fb2-446f-b2b4-a9c2792c5794-kube-api-access-5s92r\" (UniqueName: \"kubernetes.io/projected/90333f30-7fb2-446f-b2b4-a9c2792c5794-kube-api-access-5s92r\") pod \"90333f30-7fb2-446f-b2b4-a9c2792c5794\" (UID: \"90333f30-7fb2-446f-b2b4-a9c2792c5794\") " Nov 02 22:26:47.139908 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.139884 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90333f30-7fb2-446f-b2b4-a9c2792c5794-plugins-config-volume" pod "90333f30-7fb2-446f-b2b4-a9c2792c5794" (UID: "90333f30-7fb2-446f-b2b4-a9c2792c5794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:47.140748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.140723 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-openshift-ca" pod "bfb69aa7-7281-477d-9307-61d2bc250f02" (UID: "bfb69aa7-7281-477d-9307-61d2bc250f02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:47.142541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.142339 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-xks-ca" pod "bfb69aa7-7281-477d-9307-61d2bc250f02" (UID: "bfb69aa7-7281-477d-9307-61d2bc250f02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:47.142541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.142481 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb69aa7-7281-477d-9307-61d2bc250f02-kube-api-access-2tbzm" pod "bfb69aa7-7281-477d-9307-61d2bc250f02" (UID: "bfb69aa7-7281-477d-9307-61d2bc250f02"). InnerVolumeSpecName "kube-api-access-2tbzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:47.142541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.142508 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90333f30-7fb2-446f-b2b4-a9c2792c5794-kube-api-access-5s92r" pod "90333f30-7fb2-446f-b2b4-a9c2792c5794" (UID: "90333f30-7fb2-446f-b2b4-a9c2792c5794"). InnerVolumeSpecName "kube-api-access-5s92r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:47.240534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.240444 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08b0dad1-c312-47fd-ac10-8b4d26a32735-kube-api-access-24mv8\" (UniqueName: \"kubernetes.io/projected/08b0dad1-c312-47fd-ac10-8b4d26a32735-kube-api-access-24mv8\") pod \"08b0dad1-c312-47fd-ac10-8b4d26a32735\" (UID: \"08b0dad1-c312-47fd-ac10-8b4d26a32735\") " Nov 02 22:26:47.240534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.240508 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08b0dad1-c312-47fd-ac10-8b4d26a32735-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b0dad1-c312-47fd-ac10-8b4d26a32735-plugins-config-volume\") pod \"08b0dad1-c312-47fd-ac10-8b4d26a32735\" (UID: \"08b0dad1-c312-47fd-ac10-8b4d26a32735\") " Nov 02 22:26:47.240912 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.240889 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b0dad1-c312-47fd-ac10-8b4d26a32735-plugins-config-volume" pod "08b0dad1-c312-47fd-ac10-8b4d26a32735" (UID: "08b0dad1-c312-47fd-ac10-8b4d26a32735"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:47.241016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.241001 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08b0dad1-c312-47fd-ac10-8b4d26a32735-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.241079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.241018 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90333f30-7fb2-446f-b2b4-a9c2792c5794-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.241079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.241028 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s92r\" (UniqueName: \"kubernetes.io/projected/90333f30-7fb2-446f-b2b4-a9c2792c5794-kube-api-access-5s92r\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.241079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.241038 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tbzm\" (UniqueName: \"kubernetes.io/projected/bfb69aa7-7281-477d-9307-61d2bc250f02-kube-api-access-2tbzm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.241079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.241048 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.241079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.241058 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb69aa7-7281-477d-9307-61d2bc250f02-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.242686 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.242660 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b0dad1-c312-47fd-ac10-8b4d26a32735-kube-api-access-24mv8" pod "08b0dad1-c312-47fd-ac10-8b4d26a32735" (UID: "08b0dad1-c312-47fd-ac10-8b4d26a32735"). InnerVolumeSpecName "kube-api-access-24mv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:47.342005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.341967 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24mv8\" (UniqueName: \"kubernetes.io/projected/08b0dad1-c312-47fd-ac10-8b4d26a32735-kube-api-access-24mv8\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.632716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.632559 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:26:47.653770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.652710 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-credential-socket\") pod \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " Nov 02 22:26:47.653770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.652806 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-certs\") pod \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " Nov 02 22:26:47.653770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.652837 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istiod-ca-cert\") pod \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " Nov 02 22:26:47.653770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.652870 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-envoy\") pod \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " Nov 02 22:26:47.653770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.652962 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-podinfo\") pod \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " Nov 02 22:26:47.653770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.653002 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-socket\") pod \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " Nov 02 22:26:47.653770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.653074 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-data\") pod \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " Nov 02 22:26:47.653770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.653136 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-kube-api-access-m78gw\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-kube-api-access-m78gw\") pod \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " Nov 02 22:26:47.653770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.653204 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-token\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-token\") pod \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\" (UID: \"5ec22cb4-0c48-427d-af9a-8ebc8607be17\") " Nov 02 22:26:47.658880 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.658762 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-certs" pod "5ec22cb4-0c48-427d-af9a-8ebc8607be17" (UID: "5ec22cb4-0c48-427d-af9a-8ebc8607be17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:47.659037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.658970 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-token" pod "5ec22cb4-0c48-427d-af9a-8ebc8607be17" (UID: "5ec22cb4-0c48-427d-af9a-8ebc8607be17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:47.659037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.658980 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-credential-socket" pod "5ec22cb4-0c48-427d-af9a-8ebc8607be17" (UID: "5ec22cb4-0c48-427d-af9a-8ebc8607be17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:47.659187 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.659163 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-data" pod "5ec22cb4-0c48-427d-af9a-8ebc8607be17" (UID: "5ec22cb4-0c48-427d-af9a-8ebc8607be17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:47.659241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.659176 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-socket" pod "5ec22cb4-0c48-427d-af9a-8ebc8607be17" (UID: "5ec22cb4-0c48-427d-af9a-8ebc8607be17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:47.659512 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.659416 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istiod-ca-cert" pod "5ec22cb4-0c48-427d-af9a-8ebc8607be17" (UID: "5ec22cb4-0c48-427d-af9a-8ebc8607be17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:47.667721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.661223 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-envoy" pod "5ec22cb4-0c48-427d-af9a-8ebc8607be17" (UID: "5ec22cb4-0c48-427d-af9a-8ebc8607be17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:47.667721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.662057 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-podinfo" pod "5ec22cb4-0c48-427d-af9a-8ebc8607be17" (UID: "5ec22cb4-0c48-427d-af9a-8ebc8607be17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:47.667721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.662364 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-kube-api-access-m78gw" pod "5ec22cb4-0c48-427d-af9a-8ebc8607be17" (UID: "5ec22cb4-0c48-427d-af9a-8ebc8607be17"). InnerVolumeSpecName "kube-api-access-m78gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:47.754459 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.754426 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.754459 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.754454 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.754459 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.754466 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.754724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.754476 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.754724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.754485 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.754724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.754493 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.754724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.754503 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m78gw\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-kube-api-access-m78gw\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.754724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.754512 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec22cb4-0c48-427d-af9a-8ebc8607be17-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.754724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.754520 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec22cb4-0c48-427d-af9a-8ebc8607be17-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:47.845959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.845926 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" Nov 02 22:26:47.845959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.845956 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7" event={"ID":"b4e82e0c-d7b3-4881-918d-591dc346c9e2","Type":"ContainerDied","Data":"3b27f59d3341cce860ce8f69f921dba0b14f7259f9d1f371b14d46dd3ca4a07e"} Nov 02 22:26:47.846391 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.846019 4309 scope.go:122] "RemoveContainer" containerID="ae59824fadb39d0f9e385e87350553b244a2c3c4165461189ebd39e8942e3bb8" Nov 02 22:26:47.847986 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.847954 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" Nov 02 22:26:47.847986 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.847961 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2" event={"ID":"90333f30-7fb2-446f-b2b4-a9c2792c5794","Type":"ContainerDied","Data":"1260b1263c4b46b6e53402613eea239ff2a15407c030a32158a40ca890d010ca"} Nov 02 22:26:47.849942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.849905 4309 generic.go:356] "Generic (PLEG): container finished" podID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerID="f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778" exitCode=0 Nov 02 22:26:47.850092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.850073 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" event={"ID":"5ec22cb4-0c48-427d-af9a-8ebc8607be17","Type":"ContainerDied","Data":"f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778"} Nov 02 22:26:47.850194 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.850120 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" event={"ID":"5ec22cb4-0c48-427d-af9a-8ebc8607be17","Type":"ContainerDied","Data":"fe6a8dea249eca881e20ee6efa55e238d0299b277fab331c9e177df3301f849e"} Nov 02 22:26:47.850265 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.850200 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj" Nov 02 22:26:47.851815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.851794 4309 generic.go:356] "Generic (PLEG): container finished" podID="08b0dad1-c312-47fd-ac10-8b4d26a32735" containerID="f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4" exitCode=0 Nov 02 22:26:47.851926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.851894 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" event={"ID":"08b0dad1-c312-47fd-ac10-8b4d26a32735","Type":"ContainerDied","Data":"f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4"} Nov 02 22:26:47.851926 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.851923 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" event={"ID":"08b0dad1-c312-47fd-ac10-8b4d26a32735","Type":"ContainerDied","Data":"a817287dad1cf7fa2edceb42143dfb8692647ee2d1fc151406b27e4a7f72bf88"} Nov 02 22:26:47.852046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.851978 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5" Nov 02 22:26:47.858872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.858178 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" Nov 02 22:26:47.858872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.858214 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr" event={"ID":"bfb69aa7-7281-477d-9307-61d2bc250f02","Type":"ContainerDied","Data":"5c99d006bc869e37071c6602329598d56dd316992109748f92228b65fb8e3f15"} Nov 02 22:26:47.858872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.858254 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c99d006bc869e37071c6602329598d56dd316992109748f92228b65fb8e3f15" Nov 02 22:26:47.858872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.858357 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm" Nov 02 22:26:47.862001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.861985 4309 scope.go:122] "RemoveContainer" containerID="bc999d2836b4b27856bf0c2d42870cdbce8604109b7fb1fd2747762c838786e1" Nov 02 22:26:47.878514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.878497 4309 scope.go:122] "RemoveContainer" containerID="f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778" Nov 02 22:26:47.887722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.887699 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj"] Nov 02 22:26:47.892450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.892430 4309 scope.go:122] "RemoveContainer" containerID="f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778" Nov 02 22:26:47.892793 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:47.892769 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778\": container with ID starting with f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778 not found: ID does not exist" containerID="f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778" Nov 02 22:26:47.892867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.892807 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778"} err="failed to get container status \"f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778\": rpc error: code = NotFound desc = could not find container \"f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778\": container with ID starting with f05cc86e9c9b7d26c9b562bb3f4575797cca43949ec34ca861fff6bf8795f778 not found: ID does not exist" Nov 02 22:26:47.892867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.892826 4309 scope.go:122] "RemoveContainer" containerID="f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4" Nov 02 22:26:47.892993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.892979 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2532af-openshift-default-d47659495-9mxvj"] Nov 02 22:26:47.902845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.902824 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2"] Nov 02 22:26:47.904809 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.904791 4309 scope.go:122] "RemoveContainer" containerID="f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4" Nov 02 22:26:47.905109 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:47.905082 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4\": container with ID starting with f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4 not found: ID does not exist" containerID="f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4" Nov 02 22:26:47.905186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.905120 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4"} err="failed to get container status \"f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4\": rpc error: code = NotFound desc = could not find container \"f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4\": container with ID starting with f39f42c89249a404518943d0d5b87a6b28126612f835d0b3241ca90441648dd4 not found: ID does not exist" Nov 02 22:26:47.907363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.907344 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2532af-5fbfd46989-88ms2"] Nov 02 22:26:47.929532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.929503 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7"] Nov 02 22:26:47.934894 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.933717 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-jlbw7"] Nov 02 22:26:47.937276 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.937249 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5"] Nov 02 22:26:47.940208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.940183 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2532af-78fcf7c57f-422l5"] Nov 02 22:26:47.948371 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.948351 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm"] Nov 02 22:26:47.953983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:47.953963 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-310bb3-openshift-default-559d54dd7f-l9ljm"] Nov 02 22:26:48.191663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:48.191590 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08b0dad1-c312-47fd-ac10-8b4d26a32735" path="/var/lib/kubelet/pods/08b0dad1-c312-47fd-ac10-8b4d26a32735/volumes" Nov 02 22:26:48.192245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:48.192226 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" path="/var/lib/kubelet/pods/5ec22cb4-0c48-427d-af9a-8ebc8607be17/volumes" Nov 02 22:26:48.192766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:48.192746 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90333f30-7fb2-446f-b2b4-a9c2792c5794" path="/var/lib/kubelet/pods/90333f30-7fb2-446f-b2b4-a9c2792c5794/volumes" Nov 02 22:26:48.193768 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:48.193748 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" path="/var/lib/kubelet/pods/b4e82e0c-d7b3-4881-918d-591dc346c9e2/volumes" Nov 02 22:26:48.194328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:48.194312 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" path="/var/lib/kubelet/pods/d82d2ee2-8555-429a-968b-eb6bb6a1f2c0/volumes" Nov 02 22:26:48.476183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:48.476103 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42"] Nov 02 22:26:52.687143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.687118 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:26:52.811835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.811793 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-socket\") pod \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " Nov 02 22:26:52.812037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.811849 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-envoy\") pod \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " Nov 02 22:26:52.812037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.811893 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-kube-api-access-87rrb\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-kube-api-access-87rrb\") pod \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " Nov 02 22:26:52.812037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.811929 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istiod-ca-cert\") pod \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " Nov 02 22:26:52.812037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812015 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-certs\") pod \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " Nov 02 22:26:52.812278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812043 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-data\") pod \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " Nov 02 22:26:52.812278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812097 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-podinfo\") pod \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " Nov 02 22:26:52.812278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812110 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-socket" pod "04f9e420-c9ca-4a32-bf79-6d6d62097e9a" (UID: "04f9e420-c9ca-4a32-bf79-6d6d62097e9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:52.812278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812138 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-token\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-token\") pod \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " Nov 02 22:26:52.812278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812192 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-credential-socket\") pod \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\" (UID: \"04f9e420-c9ca-4a32-bf79-6d6d62097e9a\") " Nov 02 22:26:52.812550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812424 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-certs" pod "04f9e420-c9ca-4a32-bf79-6d6d62097e9a" (UID: "04f9e420-c9ca-4a32-bf79-6d6d62097e9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:52.812550 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812528 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istiod-ca-cert" pod "04f9e420-c9ca-4a32-bf79-6d6d62097e9a" (UID: "04f9e420-c9ca-4a32-bf79-6d6d62097e9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:52.812714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812643 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-data" pod "04f9e420-c9ca-4a32-bf79-6d6d62097e9a" (UID: "04f9e420-c9ca-4a32-bf79-6d6d62097e9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:52.812867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.812752 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-credential-socket" pod "04f9e420-c9ca-4a32-bf79-6d6d62097e9a" (UID: "04f9e420-c9ca-4a32-bf79-6d6d62097e9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:52.813132 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.813110 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.813232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.813135 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.813232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.813149 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.813232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.813162 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.813232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.813175 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.815057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.815026 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-token" pod "04f9e420-c9ca-4a32-bf79-6d6d62097e9a" (UID: "04f9e420-c9ca-4a32-bf79-6d6d62097e9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:52.815057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.815043 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-envoy" pod "04f9e420-c9ca-4a32-bf79-6d6d62097e9a" (UID: "04f9e420-c9ca-4a32-bf79-6d6d62097e9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:26:52.815191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.815064 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-podinfo" pod "04f9e420-c9ca-4a32-bf79-6d6d62097e9a" (UID: "04f9e420-c9ca-4a32-bf79-6d6d62097e9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:26:52.815191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.815180 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-kube-api-access-87rrb" pod "04f9e420-c9ca-4a32-bf79-6d6d62097e9a" (UID: "04f9e420-c9ca-4a32-bf79-6d6d62097e9a"). InnerVolumeSpecName "kube-api-access-87rrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:52.904263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.904230 4309 generic.go:356] "Generic (PLEG): container finished" podID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerID="ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5" exitCode=0 Nov 02 22:26:52.904448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.904300 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" Nov 02 22:26:52.904448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.904315 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" event={"ID":"04f9e420-c9ca-4a32-bf79-6d6d62097e9a","Type":"ContainerDied","Data":"ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5"} Nov 02 22:26:52.904448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.904373 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f" event={"ID":"04f9e420-c9ca-4a32-bf79-6d6d62097e9a","Type":"ContainerDied","Data":"5ae8f8da5f3276ff4f0fdcf82963963f7e56bf6c598a1f9f712dab113a27e51c"} Nov 02 22:26:52.904448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.904398 4309 scope.go:122] "RemoveContainer" containerID="ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5" Nov 02 22:26:52.914072 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.914051 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.914072 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.914071 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.914200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.914081 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.914200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.914092 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87rrb\" (UniqueName: \"kubernetes.io/projected/04f9e420-c9ca-4a32-bf79-6d6d62097e9a-kube-api-access-87rrb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:52.914765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.914746 4309 scope.go:122] "RemoveContainer" containerID="facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5" Nov 02 22:26:52.925385 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.925368 4309 scope.go:122] "RemoveContainer" containerID="ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5" Nov 02 22:26:52.925724 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:52.925704 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5\": container with ID starting with ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5 not found: ID does not exist" containerID="ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5" Nov 02 22:26:52.925803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.925738 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5"} err="failed to get container status \"ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5\": rpc error: code = NotFound desc = could not find container \"ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5\": container with ID starting with ec3005d148fff0ea0bf2bf6123e3d81a0bcd46131bd4a4417ba9c1ceed3a52a5 not found: ID does not exist" Nov 02 22:26:52.925803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.925762 4309 scope.go:122] "RemoveContainer" containerID="facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5" Nov 02 22:26:52.926059 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:26:52.926042 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5\": container with ID starting with facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5 not found: ID does not exist" containerID="facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5" Nov 02 22:26:52.926126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.926066 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5"} err="failed to get container status \"facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5\": rpc error: code = NotFound desc = could not find container \"facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5\": container with ID starting with facde848e1411e1bf5d677c73043f9adf1fb1d3290f7ca51c230d631d30928c5 not found: ID does not exist" Nov 02 22:26:52.931370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.931348 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f"] Nov 02 22:26:52.934482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:52.934460 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a3713a-openshift-default-666f8f5f75-kgh8f"] Nov 02 22:26:53.745346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:53.745310 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg"] Nov 02 22:26:53.764259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:53.764225 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h"] Nov 02 22:26:54.187232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.187196 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" path="/var/lib/kubelet/pods/04f9e420-c9ca-4a32-bf79-6d6d62097e9a/volumes" Nov 02 22:26:54.328851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.328815 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58"] Nov 02 22:26:54.329293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329280 4309 cpu_manager.go:414] "containerMap: removing container" podUID="08b0dad1-c312-47fd-ac10-8b4d26a32735" containerName="payload-pre-processing" Nov 02 22:26:54.329340 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329293 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08b0dad1-c312-47fd-ac10-8b4d26a32735" containerName="payload-pre-processing" Nov 02 22:26:54.329340 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329310 4309 cpu_manager.go:414] "containerMap: removing container" podUID="bfb69aa7-7281-477d-9307-61d2bc250f02" containerName="revoke-keys" Nov 02 22:26:54.329340 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329315 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb69aa7-7281-477d-9307-61d2bc250f02" containerName="revoke-keys" Nov 02 22:26:54.329340 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329335 4309 cpu_manager.go:414] "containerMap: removing container" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerName="istio-proxy" Nov 02 22:26:54.329340 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329340 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329354 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329358 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329367 4309 cpu_manager.go:414] "containerMap: removing container" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329372 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329377 4309 cpu_manager.go:414] "containerMap: removing container" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329384 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329394 4309 cpu_manager.go:414] "containerMap: removing container" podUID="90333f30-7fb2-446f-b2b4-a9c2792c5794" containerName="payload-processing" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329398 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90333f30-7fb2-446f-b2b4-a9c2792c5794" containerName="payload-processing" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329412 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329416 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329477 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329486 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82d2ee2-8555-429a-968b-eb6bb6a1f2c0" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329493 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec22cb4-0c48-427d-af9a-8ebc8607be17" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329499 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f9e420-c9ca-4a32-bf79-6d6d62097e9a" containerName="istio-proxy" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329506 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="90333f30-7fb2-446f-b2b4-a9c2792c5794" containerName="payload-processing" Nov 02 22:26:54.329513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329514 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e82e0c-d7b3-4881-918d-591dc346c9e2" containerName="istio-proxy" Nov 02 22:26:54.330019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329521 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="08b0dad1-c312-47fd-ac10-8b4d26a32735" containerName="payload-pre-processing" Nov 02 22:26:54.330019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.329527 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb69aa7-7281-477d-9307-61d2bc250f02" containerName="revoke-keys" Nov 02 22:26:54.334221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.334206 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.340640 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.340589 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:26:54.341913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.341888 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:26:54.342053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.341983 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:26:54.342053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.342026 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:26:54.342173 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.342097 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58"] Nov 02 22:26:54.431856 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.431818 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nksfd\" (UniqueName: \"kubernetes.io/projected/e9614ca9-3441-43e7-a9f4-278ebff05cb6-kube-api-access-nksfd\") pod \"maas-api-revoke-sub-25396947-b2f58\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.432043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.431937 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25396947-b2f58\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.432043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.431990 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25396947-b2f58\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.533201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.533121 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25396947-b2f58\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.533201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.533162 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25396947-b2f58\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.533387 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.533225 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nksfd\" (UniqueName: \"kubernetes.io/projected/e9614ca9-3441-43e7-a9f4-278ebff05cb6-kube-api-access-nksfd\") pod \"maas-api-revoke-sub-25396947-b2f58\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.533944 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.533924 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25396947-b2f58\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.535473 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.535454 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25396947-b2f58\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.540582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.540557 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nksfd\" (UniqueName: \"kubernetes.io/projected/e9614ca9-3441-43e7-a9f4-278ebff05cb6-kube-api-access-nksfd\") pod \"maas-api-revoke-sub-25396947-b2f58\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.651232 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.651203 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:54.934706 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:54.931071 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58"] Nov 02 22:26:54.940842 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:26:54.940809 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9614ca9_3441_43e7_a9f4_278ebff05cb6.slice/crio-4645328a8dc314591b5602252ab083d08b12a3ea2634de2a7d54e3373b5fd9fc WatchSource:0}: Error finding container 4645328a8dc314591b5602252ab083d08b12a3ea2634de2a7d54e3373b5fd9fc: Status 404 returned error can't find the container with id 4645328a8dc314591b5602252ab083d08b12a3ea2634de2a7d54e3373b5fd9fc Nov 02 22:26:55.929691 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:55.929644 4309 generic.go:356] "Generic (PLEG): container finished" podID="e9614ca9-3441-43e7-a9f4-278ebff05cb6" containerID="dddba56ce3287850d05de51acf23349e5b5809dfe7d6b306421b1abb01800646" exitCode=0 Nov 02 22:26:55.929881 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:55.929738 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" event={"ID":"e9614ca9-3441-43e7-a9f4-278ebff05cb6","Type":"ContainerDied","Data":"dddba56ce3287850d05de51acf23349e5b5809dfe7d6b306421b1abb01800646"} Nov 02 22:26:55.929881 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:55.929801 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" event={"ID":"e9614ca9-3441-43e7-a9f4-278ebff05cb6","Type":"ContainerStarted","Data":"4645328a8dc314591b5602252ab083d08b12a3ea2634de2a7d54e3373b5fd9fc"} Nov 02 22:26:57.090872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.090845 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:57.163185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.163152 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-openshift-ca\") pod \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " Nov 02 22:26:57.163364 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.163256 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-xks-ca\") pod \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " Nov 02 22:26:57.163423 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.163392 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9614ca9-3441-43e7-a9f4-278ebff05cb6-kube-api-access-nksfd\" (UniqueName: \"kubernetes.io/projected/e9614ca9-3441-43e7-a9f4-278ebff05cb6-kube-api-access-nksfd\") pod \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\" (UID: \"e9614ca9-3441-43e7-a9f4-278ebff05cb6\") " Nov 02 22:26:57.163503 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.163481 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-openshift-ca" pod "e9614ca9-3441-43e7-a9f4-278ebff05cb6" (UID: "e9614ca9-3441-43e7-a9f4-278ebff05cb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:26:57.163810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.163786 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:57.165384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.165356 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-xks-ca" pod "e9614ca9-3441-43e7-a9f4-278ebff05cb6" (UID: "e9614ca9-3441-43e7-a9f4-278ebff05cb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:26:57.165490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.165410 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9614ca9-3441-43e7-a9f4-278ebff05cb6-kube-api-access-nksfd" pod "e9614ca9-3441-43e7-a9f4-278ebff05cb6" (UID: "e9614ca9-3441-43e7-a9f4-278ebff05cb6"). InnerVolumeSpecName "kube-api-access-nksfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:26:57.265146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.265074 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9614ca9-3441-43e7-a9f4-278ebff05cb6-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:57.265146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.265104 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nksfd\" (UniqueName: \"kubernetes.io/projected/e9614ca9-3441-43e7-a9f4-278ebff05cb6-kube-api-access-nksfd\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:26:57.951375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.951343 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" Nov 02 22:26:57.951375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.951355 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58" event={"ID":"e9614ca9-3441-43e7-a9f4-278ebff05cb6","Type":"ContainerDied","Data":"4645328a8dc314591b5602252ab083d08b12a3ea2634de2a7d54e3373b5fd9fc"} Nov 02 22:26:57.951560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:57.951401 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4645328a8dc314591b5602252ab083d08b12a3ea2634de2a7d54e3373b5fd9fc" Nov 02 22:26:58.746615 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:58.746576 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" podUID="50696516-c79b-4513-99c7-1175999a2e50" containerName="payload-processing" containerID="cri-o://958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8" gracePeriod=25 Nov 02 22:26:58.765181 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:58.765141 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" podUID="5b0b2263-76f6-4f5d-b402-be9d5ec3b645" containerName="payload-pre-processing" containerID="cri-o://9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d" gracePeriod=25 Nov 02 22:26:59.179383 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:26:59.179346 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" podUID="5b0b2263-76f6-4f5d-b402-be9d5ec3b645" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:27:00.259146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:00.259110 4309 scope.go:122] "RemoveContainer" containerID="a0ac5e26ec29ea4c810fc2cdc05cc281bd3981da2ddc9e95fbaccf6a662b5c45" Nov 02 22:27:03.496497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.496435 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" containerID="cri-o://6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191" gracePeriod=45 Nov 02 22:27:03.754497 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.754418 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:27:03.837015 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.836985 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a7688c-e69a-4daa-b742-fa4996d2179c-kube-api-access-9ljkj\" (UniqueName: \"kubernetes.io/projected/62a7688c-e69a-4daa-b742-fa4996d2179c-kube-api-access-9ljkj\") pod \"62a7688c-e69a-4daa-b742-fa4996d2179c\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " Nov 02 22:27:03.837199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.837031 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-dshm\") pod \"62a7688c-e69a-4daa-b742-fa4996d2179c\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " Nov 02 22:27:03.837199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.837065 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-tmp-dir\") pod \"62a7688c-e69a-4daa-b742-fa4996d2179c\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " Nov 02 22:27:03.837199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.837103 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-model-cache\") pod \"62a7688c-e69a-4daa-b742-fa4996d2179c\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " Nov 02 22:27:03.837199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.837143 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-home\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-home\") pod \"62a7688c-e69a-4daa-b742-fa4996d2179c\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " Nov 02 22:27:03.837433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.837219 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a7688c-e69a-4daa-b742-fa4996d2179c-tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7688c-e69a-4daa-b742-fa4996d2179c-tls-certs\") pod \"62a7688c-e69a-4daa-b742-fa4996d2179c\" (UID: \"62a7688c-e69a-4daa-b742-fa4996d2179c\") " Nov 02 22:27:03.837433 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.837366 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-tmp-dir" pod "62a7688c-e69a-4daa-b742-fa4996d2179c" (UID: "62a7688c-e69a-4daa-b742-fa4996d2179c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:03.837539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.837519 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-home" pod "62a7688c-e69a-4daa-b742-fa4996d2179c" (UID: "62a7688c-e69a-4daa-b742-fa4996d2179c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:03.837612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.837590 4309 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-tmp-dir\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:03.837705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.837634 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-model-cache" pod "62a7688c-e69a-4daa-b742-fa4996d2179c" (UID: "62a7688c-e69a-4daa-b742-fa4996d2179c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:03.839372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.839348 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-dshm" pod "62a7688c-e69a-4daa-b742-fa4996d2179c" (UID: "62a7688c-e69a-4daa-b742-fa4996d2179c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:03.839654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.839632 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a7688c-e69a-4daa-b742-fa4996d2179c-tls-certs" pod "62a7688c-e69a-4daa-b742-fa4996d2179c" (UID: "62a7688c-e69a-4daa-b742-fa4996d2179c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:03.839771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.839660 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a7688c-e69a-4daa-b742-fa4996d2179c-kube-api-access-9ljkj" pod "62a7688c-e69a-4daa-b742-fa4996d2179c" (UID: "62a7688c-e69a-4daa-b742-fa4996d2179c"). InnerVolumeSpecName "kube-api-access-9ljkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:03.938862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.938829 4309 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-model-cache\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:03.938862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.938857 4309 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-home\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:03.938862 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.938867 4309 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a7688c-e69a-4daa-b742-fa4996d2179c-tls-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:03.939106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.938875 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ljkj\" (UniqueName: \"kubernetes.io/projected/62a7688c-e69a-4daa-b742-fa4996d2179c-kube-api-access-9ljkj\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:03.939106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.938885 4309 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62a7688c-e69a-4daa-b742-fa4996d2179c-dshm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:03.994768 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.994738 4309 generic.go:356] "Generic (PLEG): container finished" podID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerID="6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191" exitCode=0 Nov 02 22:27:03.994904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.994784 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" Nov 02 22:27:03.994904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.994816 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" event={"ID":"62a7688c-e69a-4daa-b742-fa4996d2179c","Type":"ContainerDied","Data":"6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191"} Nov 02 22:27:03.994904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.994861 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42" event={"ID":"62a7688c-e69a-4daa-b742-fa4996d2179c","Type":"ContainerDied","Data":"4e087cc35f131971db2945919cd42897b64a5b7e2540cefa5701f4f4c2204cae"} Nov 02 22:27:03.994904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:03.994880 4309 scope.go:122] "RemoveContainer" containerID="6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191" Nov 02 22:27:04.005126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:04.005107 4309 scope.go:122] "RemoveContainer" containerID="6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191" Nov 02 22:27:04.005394 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:27:04.005373 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191\": container with ID starting with 6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191 not found: ID does not exist" containerID="6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191" Nov 02 22:27:04.005470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:04.005408 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191"} err="failed to get container status \"6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191\": rpc error: code = NotFound desc = could not find container \"6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191\": container with ID starting with 6c2773c4a514a2ae384846add95aac6332b961017cca7bb7ffe2cee70a073191 not found: ID does not exist" Nov 02 22:27:04.017480 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:04.017442 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42"] Nov 02 22:27:04.020798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:04.020776 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-361a44/premium-w5-361a44-kserve-5f6bb68996-fhm42"] Nov 02 22:27:04.187899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:04.187867 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" path="/var/lib/kubelet/pods/62a7688c-e69a-4daa-b742-fa4996d2179c/volumes" Nov 02 22:27:08.162354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:08.162258 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" podUID="50696516-c79b-4513-99c7-1175999a2e50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:27:09.178754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:09.178716 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" podUID="5b0b2263-76f6-4f5d-b402-be9d5ec3b645" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:27:09.563287 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:09.563250 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr"] Nov 02 22:27:09.566471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:09.566446 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-361a44-mqtbr"] Nov 02 22:27:10.189371 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:10.189335 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb69aa7-7281-477d-9307-61d2bc250f02" path="/var/lib/kubelet/pods/bfb69aa7-7281-477d-9307-61d2bc250f02/volumes" Nov 02 22:27:11.607633 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:27:11.607587 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:27:16.487527 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:16.487482 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z"] Nov 02 22:27:16.488034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:16.487740 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" containerID="cri-o://bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5" gracePeriod=30 Nov 02 22:27:16.492363 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:16.492335 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck"] Nov 02 22:27:16.493667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:16.493637 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" containerID="cri-o://2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5" gracePeriod=30 Nov 02 22:27:16.976286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:16.976251 4309 patch_prober.go:29] interesting pod/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 22:27:16.976478 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:16.976315 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 22:27:18.163066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:18.163020 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" podUID="50696516-c79b-4513-99c7-1175999a2e50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 22:27:19.027995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.027972 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:27:19.059552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.059529 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:27:19.134753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.134712 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-plugins-config-volume\") pod \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\" (UID: \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\") " Nov 02 22:27:19.134947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.134813 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-kube-api-access-6j8x7\" (UniqueName: \"kubernetes.io/projected/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-kube-api-access-6j8x7\") pod \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\" (UID: \"5b0b2263-76f6-4f5d-b402-be9d5ec3b645\") " Nov 02 22:27:19.134947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.134870 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50696516-c79b-4513-99c7-1175999a2e50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50696516-c79b-4513-99c7-1175999a2e50-plugins-config-volume\") pod \"50696516-c79b-4513-99c7-1175999a2e50\" (UID: \"50696516-c79b-4513-99c7-1175999a2e50\") " Nov 02 22:27:19.134947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.134925 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50696516-c79b-4513-99c7-1175999a2e50-kube-api-access-zn4qd\" (UniqueName: \"kubernetes.io/projected/50696516-c79b-4513-99c7-1175999a2e50-kube-api-access-zn4qd\") pod \"50696516-c79b-4513-99c7-1175999a2e50\" (UID: \"50696516-c79b-4513-99c7-1175999a2e50\") " Nov 02 22:27:19.135265 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.135105 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-plugins-config-volume" pod "5b0b2263-76f6-4f5d-b402-be9d5ec3b645" (UID: "5b0b2263-76f6-4f5d-b402-be9d5ec3b645"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:19.135348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.135272 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50696516-c79b-4513-99c7-1175999a2e50-plugins-config-volume" pod "50696516-c79b-4513-99c7-1175999a2e50" (UID: "50696516-c79b-4513-99c7-1175999a2e50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:19.137005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.136981 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-kube-api-access-6j8x7" pod "5b0b2263-76f6-4f5d-b402-be9d5ec3b645" (UID: "5b0b2263-76f6-4f5d-b402-be9d5ec3b645"). InnerVolumeSpecName "kube-api-access-6j8x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:19.137141 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.137019 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50696516-c79b-4513-99c7-1175999a2e50-kube-api-access-zn4qd" pod "50696516-c79b-4513-99c7-1175999a2e50" (UID: "50696516-c79b-4513-99c7-1175999a2e50"). InnerVolumeSpecName "kube-api-access-zn4qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:19.147018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.146994 4309 generic.go:356] "Generic (PLEG): container finished" podID="5b0b2263-76f6-4f5d-b402-be9d5ec3b645" containerID="9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d" exitCode=0 Nov 02 22:27:19.147144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.147043 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" Nov 02 22:27:19.147144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.147078 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" event={"ID":"5b0b2263-76f6-4f5d-b402-be9d5ec3b645","Type":"ContainerDied","Data":"9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d"} Nov 02 22:27:19.147144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.147129 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h" event={"ID":"5b0b2263-76f6-4f5d-b402-be9d5ec3b645","Type":"ContainerDied","Data":"205a5dd3af0b4cfc40559cc7a2cb02f58c3af206b573c732c8c778b55c88c54a"} Nov 02 22:27:19.147317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.147158 4309 scope.go:122] "RemoveContainer" containerID="9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d" Nov 02 22:27:19.148767 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.148739 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" Nov 02 22:27:19.148875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.148770 4309 generic.go:356] "Generic (PLEG): container finished" podID="50696516-c79b-4513-99c7-1175999a2e50" containerID="958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8" exitCode=0 Nov 02 22:27:19.148875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.148791 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" event={"ID":"50696516-c79b-4513-99c7-1175999a2e50","Type":"ContainerDied","Data":"958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8"} Nov 02 22:27:19.148875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.148833 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg" event={"ID":"50696516-c79b-4513-99c7-1175999a2e50","Type":"ContainerDied","Data":"6040b619de01201b22d37fbeb976d2cd1d86ed22b9ad3c639295e44cea413a99"} Nov 02 22:27:19.157315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.157291 4309 scope.go:122] "RemoveContainer" containerID="9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d" Nov 02 22:27:19.157869 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:27:19.157671 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d\": container with ID starting with 9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d not found: ID does not exist" containerID="9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d" Nov 02 22:27:19.157869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.157740 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d"} err="failed to get container status \"9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d\": rpc error: code = NotFound desc = could not find container \"9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d\": container with ID starting with 9f25da70595546db5d787169de6ff2ce9473dfa26673426722a6cd8033400c6d not found: ID does not exist" Nov 02 22:27:19.157869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.157762 4309 scope.go:122] "RemoveContainer" containerID="958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8" Nov 02 22:27:19.167305 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.167283 4309 scope.go:122] "RemoveContainer" containerID="958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8" Nov 02 22:27:19.167621 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:27:19.167570 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8\": container with ID starting with 958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8 not found: ID does not exist" containerID="958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8" Nov 02 22:27:19.167621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.167592 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8"} err="failed to get container status \"958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8\": rpc error: code = NotFound desc = could not find container \"958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8\": container with ID starting with 958643ff4980cee177835311ac28fc0b3aaccffede2660f2cbdfe3245def3bd8 not found: ID does not exist" Nov 02 22:27:19.180099 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.180074 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h"] Nov 02 22:27:19.187702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.187659 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-361a44-597bb8cf75-g8g8h"] Nov 02 22:27:19.200270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.200241 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg"] Nov 02 22:27:19.204482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.204461 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-361a44-5f6f68994c-6whzg"] Nov 02 22:27:19.236713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.236665 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn4qd\" (UniqueName: \"kubernetes.io/projected/50696516-c79b-4513-99c7-1175999a2e50-kube-api-access-zn4qd\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:19.236713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.236713 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:19.236846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.236723 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j8x7\" (UniqueName: \"kubernetes.io/projected/5b0b2263-76f6-4f5d-b402-be9d5ec3b645-kube-api-access-6j8x7\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:19.236846 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:19.236732 4309 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50696516-c79b-4513-99c7-1175999a2e50-plugins-config-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:20.188513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:20.188475 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50696516-c79b-4513-99c7-1175999a2e50" path="/var/lib/kubelet/pods/50696516-c79b-4513-99c7-1175999a2e50/volumes" Nov 02 22:27:20.189088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:20.188937 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0b2263-76f6-4f5d-b402-be9d5ec3b645" path="/var/lib/kubelet/pods/5b0b2263-76f6-4f5d-b402-be9d5ec3b645/volumes" Nov 02 22:27:21.954905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:21.954863 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:27:22.069048 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.069023 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:27:22.073628 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.073591 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-data\") pod \"ed62b2b1-be33-450c-8833-ed15060827cc\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " Nov 02 22:27:22.073790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.073645 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-istio-token\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-istio-token\") pod \"ed62b2b1-be33-450c-8833-ed15060827cc\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " Nov 02 22:27:22.073790 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.073728 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-kube-api-access-f97kc\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-kube-api-access-f97kc\") pod \"ed62b2b1-be33-450c-8833-ed15060827cc\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " Nov 02 22:27:22.073940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.073811 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-certs\") pod \"ed62b2b1-be33-450c-8833-ed15060827cc\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " Nov 02 22:27:22.073940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.073870 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed62b2b1-be33-450c-8833-ed15060827cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed62b2b1-be33-450c-8833-ed15060827cc-istio-podinfo\") pod \"ed62b2b1-be33-450c-8833-ed15060827cc\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " Nov 02 22:27:22.073940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.073900 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-data" pod "ed62b2b1-be33-450c-8833-ed15060827cc" (UID: "ed62b2b1-be33-450c-8833-ed15060827cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.073940 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.073914 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed62b2b1-be33-450c-8833-ed15060827cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed62b2b1-be33-450c-8833-ed15060827cc-istiod-ca-cert\") pod \"ed62b2b1-be33-450c-8833-ed15060827cc\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " Nov 02 22:27:22.074159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.073950 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-socket\") pod \"ed62b2b1-be33-450c-8833-ed15060827cc\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " Nov 02 22:27:22.074159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074012 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-envoy\") pod \"ed62b2b1-be33-450c-8833-ed15060827cc\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " Nov 02 22:27:22.074159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074080 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-credential-socket\") pod \"ed62b2b1-be33-450c-8833-ed15060827cc\" (UID: \"ed62b2b1-be33-450c-8833-ed15060827cc\") " Nov 02 22:27:22.074159 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074090 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-certs" pod "ed62b2b1-be33-450c-8833-ed15060827cc" (UID: "ed62b2b1-be33-450c-8833-ed15060827cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.074413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074392 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed62b2b1-be33-450c-8833-ed15060827cc-istiod-ca-cert" pod "ed62b2b1-be33-450c-8833-ed15060827cc" (UID: "ed62b2b1-be33-450c-8833-ed15060827cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:22.074557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074524 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-socket" pod "ed62b2b1-be33-450c-8833-ed15060827cc" (UID: "ed62b2b1-be33-450c-8833-ed15060827cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.074700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074607 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-credential-socket" pod "ed62b2b1-be33-450c-8833-ed15060827cc" (UID: "ed62b2b1-be33-450c-8833-ed15060827cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.074700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074647 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.074700 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074663 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.074899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074706 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed62b2b1-be33-450c-8833-ed15060827cc-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.074899 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.074718 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.076807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.076762 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-kube-api-access-f97kc" pod "ed62b2b1-be33-450c-8833-ed15060827cc" (UID: "ed62b2b1-be33-450c-8833-ed15060827cc"). InnerVolumeSpecName "kube-api-access-f97kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:22.076807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.076792 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-envoy" pod "ed62b2b1-be33-450c-8833-ed15060827cc" (UID: "ed62b2b1-be33-450c-8833-ed15060827cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.076952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.076812 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-istio-token" pod "ed62b2b1-be33-450c-8833-ed15060827cc" (UID: "ed62b2b1-be33-450c-8833-ed15060827cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:22.077014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.076988 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed62b2b1-be33-450c-8833-ed15060827cc-istio-podinfo" pod "ed62b2b1-be33-450c-8833-ed15060827cc" (UID: "ed62b2b1-be33-450c-8833-ed15060827cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:22.175536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175457 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-socket\") pod \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " Nov 02 22:27:22.175536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175504 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-certs\") pod \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " Nov 02 22:27:22.175793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175556 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-podinfo\") pod \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " Nov 02 22:27:22.175793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175582 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-data\") pod \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " Nov 02 22:27:22.175793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175624 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istiod-ca-cert\") pod \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " Nov 02 22:27:22.175793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175661 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-envoy\") pod \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " Nov 02 22:27:22.175793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175715 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-token\") pod \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " Nov 02 22:27:22.175793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175750 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-credential-socket\") pod \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " Nov 02 22:27:22.175793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175787 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-kube-api-access-jj29f\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-kube-api-access-jj29f\") pod \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\" (UID: \"b1fc566d-a0b9-4072-964c-7ff08b445d4a\") " Nov 02 22:27:22.176230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175837 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-socket" pod "b1fc566d-a0b9-4072-964c-7ff08b445d4a" (UID: "b1fc566d-a0b9-4072-964c-7ff08b445d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.176230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175907 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-data" pod "b1fc566d-a0b9-4072-964c-7ff08b445d4a" (UID: "b1fc566d-a0b9-4072-964c-7ff08b445d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.176230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.175928 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-certs" pod "b1fc566d-a0b9-4072-964c-7ff08b445d4a" (UID: "b1fc566d-a0b9-4072-964c-7ff08b445d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.176230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176215 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istiod-ca-cert" pod "b1fc566d-a0b9-4072-964c-7ff08b445d4a" (UID: "b1fc566d-a0b9-4072-964c-7ff08b445d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:22.176476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176312 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.176476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176353 4309 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-data\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.176476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176370 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed62b2b1-be33-450c-8833-ed15060827cc-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.176476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176388 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.176476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176401 4309 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.176476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176416 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f97kc\" (UniqueName: \"kubernetes.io/projected/ed62b2b1-be33-450c-8833-ed15060827cc-kube-api-access-f97kc\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.176476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176433 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-credential-socket" pod "b1fc566d-a0b9-4072-964c-7ff08b445d4a" (UID: "b1fc566d-a0b9-4072-964c-7ff08b445d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.176476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176444 4309 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-workload-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.176896 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.176459 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed62b2b1-be33-450c-8833-ed15060827cc-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.178355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.178320 4309 generic.go:356] "Generic (PLEG): container finished" podID="ed62b2b1-be33-450c-8833-ed15060827cc" containerID="2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5" exitCode=0 Nov 02 22:27:22.178482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.178460 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" event={"ID":"ed62b2b1-be33-450c-8833-ed15060827cc","Type":"ContainerDied","Data":"2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5"} Nov 02 22:27:22.178564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.178505 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" Nov 02 22:27:22.178564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.178556 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck" event={"ID":"ed62b2b1-be33-450c-8833-ed15060827cc","Type":"ContainerDied","Data":"c56d914daaf2df67268e40f55b8a2405f669e542693893636b68a6072f6b3a32"} Nov 02 22:27:22.178712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.178589 4309 scope.go:122] "RemoveContainer" containerID="2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5" Nov 02 22:27:22.178712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.178616 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-podinfo" pod "b1fc566d-a0b9-4072-964c-7ff08b445d4a" (UID: "b1fc566d-a0b9-4072-964c-7ff08b445d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 22:27:22.178820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.178761 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-envoy" pod "b1fc566d-a0b9-4072-964c-7ff08b445d4a" (UID: "b1fc566d-a0b9-4072-964c-7ff08b445d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:27:22.178879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.178816 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-kube-api-access-jj29f" pod "b1fc566d-a0b9-4072-964c-7ff08b445d4a" (UID: "b1fc566d-a0b9-4072-964c-7ff08b445d4a"). InnerVolumeSpecName "kube-api-access-jj29f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:22.179019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.178996 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-token" pod "b1fc566d-a0b9-4072-964c-7ff08b445d4a" (UID: "b1fc566d-a0b9-4072-964c-7ff08b445d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:22.180428 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.180400 4309 generic.go:356] "Generic (PLEG): container finished" podID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerID="bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5" exitCode=0 Nov 02 22:27:22.180540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.180473 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" Nov 02 22:27:22.199866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.199823 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" event={"ID":"b1fc566d-a0b9-4072-964c-7ff08b445d4a","Type":"ContainerDied","Data":"bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5"} Nov 02 22:27:22.201046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.201014 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z" event={"ID":"b1fc566d-a0b9-4072-964c-7ff08b445d4a","Type":"ContainerDied","Data":"f27625b49b1469f6e8c7c74a5fbf994668ebee59a55d75e7ade0b5f5808c92ee"} Nov 02 22:27:22.249707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.249651 4309 scope.go:122] "RemoveContainer" containerID="2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5" Nov 02 22:27:22.250061 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:27:22.250040 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5\": container with ID starting with 2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5 not found: ID does not exist" containerID="2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5" Nov 02 22:27:22.250163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.250074 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5"} err="failed to get container status \"2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5\": rpc error: code = NotFound desc = could not find container \"2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5\": container with ID starting with 2acdd69f614aca40cb9f8bfbdd7a1ec5fb34e6d4bc3bfd9c1aae5674faa3d0f5 not found: ID does not exist" Nov 02 22:27:22.250163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.250099 4309 scope.go:122] "RemoveContainer" containerID="bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5" Nov 02 22:27:22.260464 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.260438 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z"] Nov 02 22:27:22.262901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.262883 4309 scope.go:122] "RemoveContainer" containerID="bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5" Nov 02 22:27:22.263202 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:27:22.263180 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5\": container with ID starting with bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5 not found: ID does not exist" containerID="bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5" Nov 02 22:27:22.263292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.263209 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5"} err="failed to get container status \"bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5\": rpc error: code = NotFound desc = could not find container \"bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5\": container with ID starting with bd486cb2e2293ac22ac4b89817b93f8da5c3565fa31994704071d866a1f530a5 not found: ID does not exist" Nov 02 22:27:22.263824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.263804 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cpq7z"] Nov 02 22:27:22.273370 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.273102 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck"] Nov 02 22:27:22.276909 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.276890 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-361a44-openshift-default-86959d977-cwpck"] Nov 02 22:27:22.277330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.277311 4309 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-credential-socket\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.277386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.277333 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj29f\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-kube-api-access-jj29f\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.277386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.277353 4309 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-podinfo\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.277386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.277362 4309 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istiod-ca-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.277386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.277371 4309 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-envoy\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:22.277386 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:22.277379 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1fc566d-a0b9-4072-964c-7ff08b445d4a-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:24.187419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:24.187386 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" path="/var/lib/kubelet/pods/b1fc566d-a0b9-4072-964c-7ff08b445d4a/volumes" Nov 02 22:27:24.187875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:24.187860 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" path="/var/lib/kubelet/pods/ed62b2b1-be33-450c-8833-ed15060827cc/volumes" Nov 02 22:27:36.087853 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.087813 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz"] Nov 02 22:27:36.088460 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088438 4309 cpu_manager.go:414] "containerMap: removing container" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" Nov 02 22:27:36.088567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088462 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" Nov 02 22:27:36.088567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088491 4309 cpu_manager.go:414] "containerMap: removing container" podUID="e9614ca9-3441-43e7-a9f4-278ebff05cb6" containerName="revoke-keys" Nov 02 22:27:36.088567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088500 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9614ca9-3441-43e7-a9f4-278ebff05cb6" containerName="revoke-keys" Nov 02 22:27:36.088567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088515 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" Nov 02 22:27:36.088567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088523 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" Nov 02 22:27:36.088567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088554 4309 cpu_manager.go:414] "containerMap: removing container" podUID="5b0b2263-76f6-4f5d-b402-be9d5ec3b645" containerName="payload-pre-processing" Nov 02 22:27:36.088567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088562 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0b2263-76f6-4f5d-b402-be9d5ec3b645" containerName="payload-pre-processing" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088586 4309 cpu_manager.go:414] "containerMap: removing container" podUID="50696516-c79b-4513-99c7-1175999a2e50" containerName="payload-processing" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088594 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50696516-c79b-4513-99c7-1175999a2e50" containerName="payload-processing" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088605 4309 cpu_manager.go:414] "containerMap: removing container" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088612 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088737 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1fc566d-a0b9-4072-964c-7ff08b445d4a" containerName="istio-proxy" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088756 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0b2263-76f6-4f5d-b402-be9d5ec3b645" containerName="payload-pre-processing" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088770 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="50696516-c79b-4513-99c7-1175999a2e50" containerName="payload-processing" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088781 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed62b2b1-be33-450c-8833-ed15060827cc" containerName="istio-proxy" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088792 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a7688c-e69a-4daa-b742-fa4996d2179c" containerName="main" Nov 02 22:27:36.088977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.088805 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9614ca9-3441-43e7-a9f4-278ebff05cb6" containerName="revoke-keys" Nov 02 22:27:36.093937 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.093915 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.106465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.106436 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz"] Nov 02 22:27:36.220444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.220410 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.220619 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.220450 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.220619 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.220480 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.220619 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.220585 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8m2\" (UniqueName: \"kubernetes.io/projected/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-kube-api-access-mk8m2\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.220619 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.220602 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.220796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.220717 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.220796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.220759 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.220796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.220781 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.220901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.220816 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.321624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.321589 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.321815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.321630 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.321815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.321658 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.321815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.321716 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.321815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.321741 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.321815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.321768 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.321815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.321807 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8m2\" (UniqueName: \"kubernetes.io/projected/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-kube-api-access-mk8m2\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.322141 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.321835 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.322141 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.321919 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.322575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.322546 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-credential-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.322697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.322638 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-workload-socket\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.322771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.322647 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-workload-certs\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.322771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.322712 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-data\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.322904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.322886 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.324304 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.324284 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.324421 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.324398 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.328839 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.328816 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-istio-token\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.329074 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.329052 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8m2\" (UniqueName: \"kubernetes.io/projected/1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2-kube-api-access-mk8m2\") pod \"maas-default-gateway-openshift-default-696f47c6c-r8nbz\" (UID: \"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.409042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.408957 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:36.694620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.694589 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz"] Nov 02 22:27:36.695705 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:27:36.695665 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a56fdcc_fa46_4fba_b2d1_3b7ab8a7c9c2.slice/crio-804b25055c77791e799e974a928943f34c77234d3a9817639e24e2daaf64fd69 WatchSource:0}: Error finding container 804b25055c77791e799e974a928943f34c77234d3a9817639e24e2daaf64fd69: Status 404 returned error can't find the container with id 804b25055c77791e799e974a928943f34c77234d3a9817639e24e2daaf64fd69 Nov 02 22:27:36.699031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.698985 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:27:36.699154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.699083 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:27:36.699154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:36.699133 4309 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 02 22:27:37.280259 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:37.280223 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" event={"ID":"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2","Type":"ContainerStarted","Data":"4d9b898f262d10bbe116a405455133f8651d70df7f0e948fd5a2a096503006bf"} Nov 02 22:27:37.280659 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:37.280283 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" event={"ID":"1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2","Type":"ContainerStarted","Data":"804b25055c77791e799e974a928943f34c77234d3a9817639e24e2daaf64fd69"} Nov 02 22:27:37.315830 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:37.315775 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" podStartSLOduration=1.315758694 podStartE2EDuration="1.315758694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:37.313334365 +0000 UTC m=+1785.251447584" watchObservedRunningTime="2026-11-02 22:27:37.315758694 +0000 UTC m=+1785.253871909" Nov 02 22:27:37.410017 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:37.409976 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:37.411532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:37.411507 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r8nbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 02 22:27:37.411652 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:37.411555 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" podUID="1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 02 22:27:38.409338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:38.409294 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r8nbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 02 22:27:38.409837 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:38.409357 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" podUID="1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 02 22:27:39.409960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:39.409918 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r8nbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 02 22:27:39.410514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:39.409988 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" podUID="1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 02 22:27:40.410248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:40.410206 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r8nbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 02 22:27:40.410756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:40.410283 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" podUID="1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 02 22:27:41.409739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:41.409699 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r8nbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 02 22:27:41.409930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:41.409756 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" podUID="1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 02 22:27:41.617250 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:27:41.617210 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:27:42.409490 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:42.409448 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r8nbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 02 22:27:42.409686 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:42.409528 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" podUID="1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 02 22:27:43.410182 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:43.410144 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-r8nbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 02 22:27:43.410814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:43.410221 4309 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" podUID="1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 02 22:27:44.075379 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.075343 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s"] Nov 02 22:27:44.078997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.078979 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.088023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.083736 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:27:44.088023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.083757 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:27:44.088023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.083866 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:27:44.088631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.088600 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s"] Nov 02 22:27:44.088876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.088735 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:27:44.226233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.226191 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qbs\" (UniqueName: \"kubernetes.io/projected/4a0522ba-9931-4623-982a-c6925028c7c2-kube-api-access-t4qbs\") pod \"maas-api-revoke-sub-85237e45-rfz4s\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.226233 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.226240 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rfz4s\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.226511 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.226352 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rfz4s\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.327577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.327489 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rfz4s\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.327745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.327660 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qbs\" (UniqueName: \"kubernetes.io/projected/4a0522ba-9931-4623-982a-c6925028c7c2-kube-api-access-t4qbs\") pod \"maas-api-revoke-sub-85237e45-rfz4s\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.327745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.327738 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rfz4s\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.328494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.328462 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rfz4s\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.330277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.330240 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rfz4s\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.335924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.335893 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qbs\" (UniqueName: \"kubernetes.io/projected/4a0522ba-9931-4623-982a-c6925028c7c2-kube-api-access-t4qbs\") pod \"maas-api-revoke-sub-85237e45-rfz4s\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.401502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.401465 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:44.691522 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:44.691475 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s"] Nov 02 22:27:44.692740 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:27:44.692709 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0522ba_9931_4623_982a_c6925028c7c2.slice/crio-08316b1c0aee87da7e853933602e6d3d7142ac2857be7f2d1f1703ede12d0079 WatchSource:0}: Error finding container 08316b1c0aee87da7e853933602e6d3d7142ac2857be7f2d1f1703ede12d0079: Status 404 returned error can't find the container with id 08316b1c0aee87da7e853933602e6d3d7142ac2857be7f2d1f1703ede12d0079 Nov 02 22:27:45.128060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:45.128016 4309 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:45.128435 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:45.128393 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:45.129603 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:45.129580 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-r8nbz" Nov 02 22:27:45.360466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:45.360430 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" event={"ID":"4a0522ba-9931-4623-982a-c6925028c7c2","Type":"ContainerStarted","Data":"20e1a89e297db627b58c55345239e4c4ed65ad6c44aa28bd1cf70141831bb298"} Nov 02 22:27:45.360664 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:45.360487 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" event={"ID":"4a0522ba-9931-4623-982a-c6925028c7c2","Type":"ContainerStarted","Data":"08316b1c0aee87da7e853933602e6d3d7142ac2857be7f2d1f1703ede12d0079"} Nov 02 22:27:45.377526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:45.377479 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" podStartSLOduration=1.3774654100000001 podStartE2EDuration="1.37746541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:27:45.376037943 +0000 UTC m=+1793.314151160" watchObservedRunningTime="2026-11-02 22:27:45.37746541 +0000 UTC m=+1793.315578631" Nov 02 22:27:46.370344 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:46.370309 4309 generic.go:356] "Generic (PLEG): container finished" podID="4a0522ba-9931-4623-982a-c6925028c7c2" containerID="20e1a89e297db627b58c55345239e4c4ed65ad6c44aa28bd1cf70141831bb298" exitCode=0 Nov 02 22:27:46.370814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:46.370399 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" event={"ID":"4a0522ba-9931-4623-982a-c6925028c7c2","Type":"ContainerDied","Data":"20e1a89e297db627b58c55345239e4c4ed65ad6c44aa28bd1cf70141831bb298"} Nov 02 22:27:47.568807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.568778 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:47.664078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.663995 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-xks-ca\") pod \"4a0522ba-9931-4623-982a-c6925028c7c2\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " Nov 02 22:27:47.664078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.664037 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-openshift-ca\") pod \"4a0522ba-9931-4623-982a-c6925028c7c2\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " Nov 02 22:27:47.664311 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.664113 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0522ba-9931-4623-982a-c6925028c7c2-kube-api-access-t4qbs\" (UniqueName: \"kubernetes.io/projected/4a0522ba-9931-4623-982a-c6925028c7c2-kube-api-access-t4qbs\") pod \"4a0522ba-9931-4623-982a-c6925028c7c2\" (UID: \"4a0522ba-9931-4623-982a-c6925028c7c2\") " Nov 02 22:27:47.664517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.664487 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-openshift-ca" pod "4a0522ba-9931-4623-982a-c6925028c7c2" (UID: "4a0522ba-9931-4623-982a-c6925028c7c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:27:47.666279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.666247 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-xks-ca" pod "4a0522ba-9931-4623-982a-c6925028c7c2" (UID: "4a0522ba-9931-4623-982a-c6925028c7c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:27:47.666376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.666288 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0522ba-9931-4623-982a-c6925028c7c2-kube-api-access-t4qbs" pod "4a0522ba-9931-4623-982a-c6925028c7c2" (UID: "4a0522ba-9931-4623-982a-c6925028c7c2"). InnerVolumeSpecName "kube-api-access-t4qbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:27:47.765999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.765952 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:47.765999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.765990 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a0522ba-9931-4623-982a-c6925028c7c2-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:47.765999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:47.766004 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4qbs\" (UniqueName: \"kubernetes.io/projected/4a0522ba-9931-4623-982a-c6925028c7c2-kube-api-access-t4qbs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:27:48.384756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:48.384721 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" Nov 02 22:27:48.384952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:48.384715 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s" event={"ID":"4a0522ba-9931-4623-982a-c6925028c7c2","Type":"ContainerDied","Data":"08316b1c0aee87da7e853933602e6d3d7142ac2857be7f2d1f1703ede12d0079"} Nov 02 22:27:48.384952 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:48.384878 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08316b1c0aee87da7e853933602e6d3d7142ac2857be7f2d1f1703ede12d0079" Nov 02 22:27:54.021045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:54.021003 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2"] Nov 02 22:27:54.023874 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:54.023847 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4703b472-f9qp2"] Nov 02 22:27:54.187615 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:54.187583 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ace8dd-4664-431b-a1db-29ad4031a4b1" path="/var/lib/kubelet/pods/20ace8dd-4664-431b-a1db-29ad4031a4b1/volumes" Nov 02 22:27:55.020753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:55.020723 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh"] Nov 02 22:27:55.023225 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:55.023204 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd17032-r84kh"] Nov 02 22:27:56.186845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:56.186813 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68cb7024-a457-4bb4-8296-180cd1ee4617" path="/var/lib/kubelet/pods/68cb7024-a457-4bb4-8296-180cd1ee4617/volumes" Nov 02 22:27:56.514217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:56.514138 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:27:56.521705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:27:56.521685 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:28:00.404221 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:00.404190 4309 scope.go:122] "RemoveContainer" containerID="e3ac785be847cee2583b7128bb2bd938b97bedf4db189f421ef86fa08f306291" Nov 02 22:28:00.413796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:00.413773 4309 scope.go:122] "RemoveContainer" containerID="f8fd4f230d79a660671807852b75334fb035efd2ec2d80380843b54f747a687b" Nov 02 22:28:11.594817 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:28:11.594772 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:28:36.211238 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.211163 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb"] Nov 02 22:28:36.211714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.211577 4309 cpu_manager.go:414] "containerMap: removing container" podUID="4a0522ba-9931-4623-982a-c6925028c7c2" containerName="revoke-keys" Nov 02 22:28:36.211714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.211588 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0522ba-9931-4623-982a-c6925028c7c2" containerName="revoke-keys" Nov 02 22:28:36.211714 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.211692 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0522ba-9931-4623-982a-c6925028c7c2" containerName="revoke-keys" Nov 02 22:28:36.215166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.215137 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.219756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.219733 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:28:36.220000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.219934 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:28:36.220129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.220068 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:28:36.220263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.220235 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:28:36.222198 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.222167 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb"] Nov 02 22:28:36.292013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.291985 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9d8zb\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.292205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.292018 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9d8zb\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.292205 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.292047 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgj7v\" (UniqueName: \"kubernetes.io/projected/d2464b6f-1517-42dd-8578-fecc2e76c955-kube-api-access-kgj7v\") pod \"maas-api-revoke-sub-c1e271a9-9d8zb\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.393334 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.393300 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9d8zb\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.393334 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.393333 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9d8zb\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.393519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.393356 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgj7v\" (UniqueName: \"kubernetes.io/projected/d2464b6f-1517-42dd-8578-fecc2e76c955-kube-api-access-kgj7v\") pod \"maas-api-revoke-sub-c1e271a9-9d8zb\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.394062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.394036 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-9d8zb\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.395627 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.395603 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-9d8zb\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.401147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.401127 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgj7v\" (UniqueName: \"kubernetes.io/projected/d2464b6f-1517-42dd-8578-fecc2e76c955-kube-api-access-kgj7v\") pod \"maas-api-revoke-sub-c1e271a9-9d8zb\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.530644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.530566 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:36.807164 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:36.807140 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb"] Nov 02 22:28:36.808653 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:28:36.808621 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2464b6f_1517_42dd_8578_fecc2e76c955.slice/crio-b2756af155c770c61af5f3b70b2b4aac90140fd0873d2167ff20c4a780eb4554 WatchSource:0}: Error finding container b2756af155c770c61af5f3b70b2b4aac90140fd0873d2167ff20c4a780eb4554: Status 404 returned error can't find the container with id b2756af155c770c61af5f3b70b2b4aac90140fd0873d2167ff20c4a780eb4554 Nov 02 22:28:37.711520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:37.711483 4309 generic.go:356] "Generic (PLEG): container finished" podID="d2464b6f-1517-42dd-8578-fecc2e76c955" containerID="2809fb6bb89cb7a3e2a5d0890f5e9c3d38df03a0ff160e9dbec441d1b0510610" exitCode=0 Nov 02 22:28:37.711900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:37.711533 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" event={"ID":"d2464b6f-1517-42dd-8578-fecc2e76c955","Type":"ContainerDied","Data":"2809fb6bb89cb7a3e2a5d0890f5e9c3d38df03a0ff160e9dbec441d1b0510610"} Nov 02 22:28:37.711900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:37.711603 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" event={"ID":"d2464b6f-1517-42dd-8578-fecc2e76c955","Type":"ContainerStarted","Data":"b2756af155c770c61af5f3b70b2b4aac90140fd0873d2167ff20c4a780eb4554"} Nov 02 22:28:38.855294 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:38.855268 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:38.918610 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:38.918579 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-xks-ca\") pod \"d2464b6f-1517-42dd-8578-fecc2e76c955\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " Nov 02 22:28:38.918825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:38.918635 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2464b6f-1517-42dd-8578-fecc2e76c955-kube-api-access-kgj7v\" (UniqueName: \"kubernetes.io/projected/d2464b6f-1517-42dd-8578-fecc2e76c955-kube-api-access-kgj7v\") pod \"d2464b6f-1517-42dd-8578-fecc2e76c955\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " Nov 02 22:28:38.918825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:38.918783 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-openshift-ca\") pod \"d2464b6f-1517-42dd-8578-fecc2e76c955\" (UID: \"d2464b6f-1517-42dd-8578-fecc2e76c955\") " Nov 02 22:28:38.919105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:38.919076 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-openshift-ca" pod "d2464b6f-1517-42dd-8578-fecc2e76c955" (UID: "d2464b6f-1517-42dd-8578-fecc2e76c955"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:38.920531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:38.920510 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-xks-ca" pod "d2464b6f-1517-42dd-8578-fecc2e76c955" (UID: "d2464b6f-1517-42dd-8578-fecc2e76c955"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:38.920637 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:38.920616 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2464b6f-1517-42dd-8578-fecc2e76c955-kube-api-access-kgj7v" pod "d2464b6f-1517-42dd-8578-fecc2e76c955" (UID: "d2464b6f-1517-42dd-8578-fecc2e76c955"). InnerVolumeSpecName "kube-api-access-kgj7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:39.020034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:39.019951 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:39.020034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:39.019982 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgj7v\" (UniqueName: \"kubernetes.io/projected/d2464b6f-1517-42dd-8578-fecc2e76c955-kube-api-access-kgj7v\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:39.020034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:39.019993 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2464b6f-1517-42dd-8578-fecc2e76c955-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:39.725297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:39.725266 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" Nov 02 22:28:39.725297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:39.725279 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb" event={"ID":"d2464b6f-1517-42dd-8578-fecc2e76c955","Type":"ContainerDied","Data":"b2756af155c770c61af5f3b70b2b4aac90140fd0873d2167ff20c4a780eb4554"} Nov 02 22:28:39.725589 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:39.725321 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2756af155c770c61af5f3b70b2b4aac90140fd0873d2167ff20c4a780eb4554" Nov 02 22:28:41.594803 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:28:41.594766 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:28:55.030321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.030289 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck"] Nov 02 22:28:55.030771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.030757 4309 cpu_manager.go:414] "containerMap: removing container" podUID="d2464b6f-1517-42dd-8578-fecc2e76c955" containerName="revoke-keys" Nov 02 22:28:55.030832 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.030771 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2464b6f-1517-42dd-8578-fecc2e76c955" containerName="revoke-keys" Nov 02 22:28:55.030870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.030850 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2464b6f-1517-42dd-8578-fecc2e76c955" containerName="revoke-keys" Nov 02 22:28:55.034039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.034017 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.038950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.038864 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:28:55.039366 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.039326 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:28:55.040499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.040478 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:28:55.040663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.040584 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:28:55.041600 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.041576 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck"] Nov 02 22:28:55.123431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.123401 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4sck\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.123611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.123437 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cqp8\" (UniqueName: \"kubernetes.io/projected/9bf30a29-154a-406d-b4d9-28cac1305e80-kube-api-access-5cqp8\") pod \"maas-api-revoke-sub-2e1f5b5f-g4sck\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.123611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.123468 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4sck\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.224543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.224512 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4sck\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.224543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.224546 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cqp8\" (UniqueName: \"kubernetes.io/projected/9bf30a29-154a-406d-b4d9-28cac1305e80-kube-api-access-5cqp8\") pod \"maas-api-revoke-sub-2e1f5b5f-g4sck\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.224769 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.224574 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4sck\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.225278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.225258 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4sck\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.226852 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.226834 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g4sck\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.232365 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.232342 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cqp8\" (UniqueName: \"kubernetes.io/projected/9bf30a29-154a-406d-b4d9-28cac1305e80-kube-api-access-5cqp8\") pod \"maas-api-revoke-sub-2e1f5b5f-g4sck\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.350661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.350630 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:55.624416 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.624386 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck"] Nov 02 22:28:55.625807 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:28:55.625782 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf30a29_154a_406d_b4d9_28cac1305e80.slice/crio-9d87f6793ca25a8838071ab3296323a61d25202760e56e3e3460c6b0c9e0cbde WatchSource:0}: Error finding container 9d87f6793ca25a8838071ab3296323a61d25202760e56e3e3460c6b0c9e0cbde: Status 404 returned error can't find the container with id 9d87f6793ca25a8838071ab3296323a61d25202760e56e3e3460c6b0c9e0cbde Nov 02 22:28:55.859262 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:55.859190 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" event={"ID":"9bf30a29-154a-406d-b4d9-28cac1305e80","Type":"ContainerStarted","Data":"9d87f6793ca25a8838071ab3296323a61d25202760e56e3e3460c6b0c9e0cbde"} Nov 02 22:28:56.867042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:56.867001 4309 generic.go:356] "Generic (PLEG): container finished" podID="9bf30a29-154a-406d-b4d9-28cac1305e80" containerID="7120c409a45b175214e3be24b2fb6db5b95d0d6f1efb7c5129a86f210025f241" exitCode=0 Nov 02 22:28:56.867449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:56.867081 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" event={"ID":"9bf30a29-154a-406d-b4d9-28cac1305e80","Type":"ContainerDied","Data":"7120c409a45b175214e3be24b2fb6db5b95d0d6f1efb7c5129a86f210025f241"} Nov 02 22:28:58.014451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.014423 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:58.154777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.154663 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-openshift-ca\") pod \"9bf30a29-154a-406d-b4d9-28cac1305e80\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " Nov 02 22:28:58.154777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.154740 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-xks-ca\") pod \"9bf30a29-154a-406d-b4d9-28cac1305e80\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " Nov 02 22:28:58.154777 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.154770 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf30a29-154a-406d-b4d9-28cac1305e80-kube-api-access-5cqp8\" (UniqueName: \"kubernetes.io/projected/9bf30a29-154a-406d-b4d9-28cac1305e80-kube-api-access-5cqp8\") pod \"9bf30a29-154a-406d-b4d9-28cac1305e80\" (UID: \"9bf30a29-154a-406d-b4d9-28cac1305e80\") " Nov 02 22:28:58.155096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.155075 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-openshift-ca" pod "9bf30a29-154a-406d-b4d9-28cac1305e80" (UID: "9bf30a29-154a-406d-b4d9-28cac1305e80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:28:58.156849 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.156825 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-xks-ca" pod "9bf30a29-154a-406d-b4d9-28cac1305e80" (UID: "9bf30a29-154a-406d-b4d9-28cac1305e80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:28:58.156962 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.156929 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf30a29-154a-406d-b4d9-28cac1305e80-kube-api-access-5cqp8" pod "9bf30a29-154a-406d-b4d9-28cac1305e80" (UID: "9bf30a29-154a-406d-b4d9-28cac1305e80"). InnerVolumeSpecName "kube-api-access-5cqp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:28:58.256213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.256175 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:58.256213 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.256210 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf30a29-154a-406d-b4d9-28cac1305e80-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:58.256419 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.256297 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cqp8\" (UniqueName: \"kubernetes.io/projected/9bf30a29-154a-406d-b4d9-28cac1305e80-kube-api-access-5cqp8\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:28:58.882575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.882542 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" event={"ID":"9bf30a29-154a-406d-b4d9-28cac1305e80","Type":"ContainerDied","Data":"9d87f6793ca25a8838071ab3296323a61d25202760e56e3e3460c6b0c9e0cbde"} Nov 02 22:28:58.882575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.882571 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck" Nov 02 22:28:58.882814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:28:58.882589 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d87f6793ca25a8838071ab3296323a61d25202760e56e3e3460c6b0c9e0cbde" Nov 02 22:29:00.463786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:00.463755 4309 scope.go:122] "RemoveContainer" containerID="b4b0e479a6311a252e928af8c0c94d675cbf1144079b23c3eea0786db457e9b4" Nov 02 22:29:00.473987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:00.473959 4309 scope.go:122] "RemoveContainer" containerID="9db8920a7d60a1cf2e90e437b27a222977f2cac111630bd87267232963c0fa14" Nov 02 22:29:00.482872 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:00.482849 4309 scope.go:122] "RemoveContainer" containerID="4a1fd264b475d6736f0510f5f2f1d213357db4a0d1ee08cecf53ff1772c368e3" Nov 02 22:29:06.061070 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.061034 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr"] Nov 02 22:29:06.061531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.061515 4309 cpu_manager.go:414] "containerMap: removing container" podUID="9bf30a29-154a-406d-b4d9-28cac1305e80" containerName="revoke-keys" Nov 02 22:29:06.061531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.061532 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf30a29-154a-406d-b4d9-28cac1305e80" containerName="revoke-keys" Nov 02 22:29:06.061656 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.061632 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf30a29-154a-406d-b4d9-28cac1305e80" containerName="revoke-keys" Nov 02 22:29:06.064917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.064892 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.072195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.072112 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:29:06.072195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.072141 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:29:06.072195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.072168 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:06.072540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.072283 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:29:06.095810 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.095773 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr"] Nov 02 22:29:06.135282 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.135241 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-5clmr\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.135461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.135368 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpt8x\" (UniqueName: \"kubernetes.io/projected/203decfc-a23e-4b98-a63a-992e8f16e2d7-kube-api-access-jpt8x\") pod \"maas-api-revoke-sub-ec843018-5clmr\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.135461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.135433 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-5clmr\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.236499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.236463 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-5clmr\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.236734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.236533 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpt8x\" (UniqueName: \"kubernetes.io/projected/203decfc-a23e-4b98-a63a-992e8f16e2d7-kube-api-access-jpt8x\") pod \"maas-api-revoke-sub-ec843018-5clmr\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.236734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.236581 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-5clmr\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.237343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.237315 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-5clmr\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.239126 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.239096 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-5clmr\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.244027 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.244003 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpt8x\" (UniqueName: \"kubernetes.io/projected/203decfc-a23e-4b98-a63a-992e8f16e2d7-kube-api-access-jpt8x\") pod \"maas-api-revoke-sub-ec843018-5clmr\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.393380 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.393292 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:06.668346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.668316 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr"] Nov 02 22:29:06.669459 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:29:06.669435 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203decfc_a23e_4b98_a63a_992e8f16e2d7.slice/crio-5296bce1ad71f653592635829ab541f089395bf0ca6777bb532404897f2a042e WatchSource:0}: Error finding container 5296bce1ad71f653592635829ab541f089395bf0ca6777bb532404897f2a042e: Status 404 returned error can't find the container with id 5296bce1ad71f653592635829ab541f089395bf0ca6777bb532404897f2a042e Nov 02 22:29:06.937218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:06.937135 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" event={"ID":"203decfc-a23e-4b98-a63a-992e8f16e2d7","Type":"ContainerStarted","Data":"5296bce1ad71f653592635829ab541f089395bf0ca6777bb532404897f2a042e"} Nov 02 22:29:07.949462 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:07.949425 4309 generic.go:356] "Generic (PLEG): container finished" podID="203decfc-a23e-4b98-a63a-992e8f16e2d7" containerID="96c5472b28dc53cc721b69f88c21e5e29bd6bfa3d6bb09582def803599fe9d64" exitCode=0 Nov 02 22:29:07.949982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:07.949513 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" event={"ID":"203decfc-a23e-4b98-a63a-992e8f16e2d7","Type":"ContainerDied","Data":"96c5472b28dc53cc721b69f88c21e5e29bd6bfa3d6bb09582def803599fe9d64"} Nov 02 22:29:09.542724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.531730 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:09.688698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.687990 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-xks-ca\") pod \"203decfc-a23e-4b98-a63a-992e8f16e2d7\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " Nov 02 22:29:09.688698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.688054 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/203decfc-a23e-4b98-a63a-992e8f16e2d7-kube-api-access-jpt8x\" (UniqueName: \"kubernetes.io/projected/203decfc-a23e-4b98-a63a-992e8f16e2d7-kube-api-access-jpt8x\") pod \"203decfc-a23e-4b98-a63a-992e8f16e2d7\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " Nov 02 22:29:09.688698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.688232 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-openshift-ca\") pod \"203decfc-a23e-4b98-a63a-992e8f16e2d7\" (UID: \"203decfc-a23e-4b98-a63a-992e8f16e2d7\") " Nov 02 22:29:09.713697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.712868 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-openshift-ca" pod "203decfc-a23e-4b98-a63a-992e8f16e2d7" (UID: "203decfc-a23e-4b98-a63a-992e8f16e2d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:09.761997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.761912 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-xks-ca" pod "203decfc-a23e-4b98-a63a-992e8f16e2d7" (UID: "203decfc-a23e-4b98-a63a-992e8f16e2d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:09.763083 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.763012 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203decfc-a23e-4b98-a63a-992e8f16e2d7-kube-api-access-jpt8x" pod "203decfc-a23e-4b98-a63a-992e8f16e2d7" (UID: "203decfc-a23e-4b98-a63a-992e8f16e2d7"). InnerVolumeSpecName "kube-api-access-jpt8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:09.791698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.791373 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:09.791698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.791414 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/203decfc-a23e-4b98-a63a-992e8f16e2d7-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:09.791698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.791434 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpt8x\" (UniqueName: \"kubernetes.io/projected/203decfc-a23e-4b98-a63a-992e8f16e2d7-kube-api-access-jpt8x\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:09.974734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.974712 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" Nov 02 22:29:09.974924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.974706 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr" event={"ID":"203decfc-a23e-4b98-a63a-992e8f16e2d7","Type":"ContainerDied","Data":"5296bce1ad71f653592635829ab541f089395bf0ca6777bb532404897f2a042e"} Nov 02 22:29:09.974924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:09.974848 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5296bce1ad71f653592635829ab541f089395bf0ca6777bb532404897f2a042e" Nov 02 22:29:11.596808 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:29:11.596771 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:29:20.047701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:20.042724 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll"] Nov 02 22:29:20.072701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:20.070962 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beac5662-nljll"] Nov 02 22:29:20.221375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:20.221334 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2bd6a4-de5a-4c4c-9266-0a43061d2528" path="/var/lib/kubelet/pods/4d2bd6a4-de5a-4c4c-9266-0a43061d2528/volumes" Nov 02 22:29:28.949314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:28.949276 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z"] Nov 02 22:29:28.949795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:28.949777 4309 cpu_manager.go:414] "containerMap: removing container" podUID="203decfc-a23e-4b98-a63a-992e8f16e2d7" containerName="revoke-keys" Nov 02 22:29:28.949848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:28.949797 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="203decfc-a23e-4b98-a63a-992e8f16e2d7" containerName="revoke-keys" Nov 02 22:29:28.949911 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:28.949902 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="203decfc-a23e-4b98-a63a-992e8f16e2d7" containerName="revoke-keys" Nov 02 22:29:28.953311 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:28.953295 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:28.957389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:28.957348 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:28.957518 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:28.957418 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:29:28.961354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:28.961329 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z"] Nov 02 22:29:29.004446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.004418 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:29:29.004607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.004453 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:29:29.036163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.036137 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mdr\" (UniqueName: \"kubernetes.io/projected/96fe7b38-8085-493d-82cb-650451925bb1-kube-api-access-v7mdr\") pod \"maas-api-revoke-sub-046e1343-tgk9z\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.036274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.036173 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tgk9z\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.036274 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.036196 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tgk9z\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.136905 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.136875 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mdr\" (UniqueName: \"kubernetes.io/projected/96fe7b38-8085-493d-82cb-650451925bb1-kube-api-access-v7mdr\") pod \"maas-api-revoke-sub-046e1343-tgk9z\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.137080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.136920 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tgk9z\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.137080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.136945 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tgk9z\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.137634 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.137607 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-tgk9z\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.139297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.139275 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-tgk9z\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.144238 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.144217 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mdr\" (UniqueName: \"kubernetes.io/projected/96fe7b38-8085-493d-82cb-650451925bb1-kube-api-access-v7mdr\") pod \"maas-api-revoke-sub-046e1343-tgk9z\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.270244 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.270174 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:29.545622 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:29.545591 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z"] Nov 02 22:29:29.546990 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:29:29.546958 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fe7b38_8085_493d_82cb_650451925bb1.slice/crio-17283b60aead2ca71761e2ac69482766da5abaf89a18b3549a46a55a805dbac9 WatchSource:0}: Error finding container 17283b60aead2ca71761e2ac69482766da5abaf89a18b3549a46a55a805dbac9: Status 404 returned error can't find the container with id 17283b60aead2ca71761e2ac69482766da5abaf89a18b3549a46a55a805dbac9 Nov 02 22:29:30.148766 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:30.148727 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" event={"ID":"96fe7b38-8085-493d-82cb-650451925bb1","Type":"ContainerStarted","Data":"210b771f8d3b3f4fcec054ca78d6b87dd048ab9c2c68c06633aa0f67e2ec5ebe"} Nov 02 22:29:30.149165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:30.148826 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" event={"ID":"96fe7b38-8085-493d-82cb-650451925bb1","Type":"ContainerStarted","Data":"17283b60aead2ca71761e2ac69482766da5abaf89a18b3549a46a55a805dbac9"} Nov 02 22:29:31.027312 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:31.027254 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" podStartSLOduration=3.027239236 podStartE2EDuration="3.027239236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:30.163969597 +0000 UTC m=+1898.102082816" watchObservedRunningTime="2026-11-02 22:29:31.027239236 +0000 UTC m=+1898.965352454" Nov 02 22:29:31.027578 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:31.027399 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9"] Nov 02 22:29:31.030228 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:31.030205 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f39394c5-87mp9"] Nov 02 22:29:31.156443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:31.156408 4309 generic.go:356] "Generic (PLEG): container finished" podID="96fe7b38-8085-493d-82cb-650451925bb1" containerID="210b771f8d3b3f4fcec054ca78d6b87dd048ab9c2c68c06633aa0f67e2ec5ebe" exitCode=0 Nov 02 22:29:31.156865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:31.156499 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" event={"ID":"96fe7b38-8085-493d-82cb-650451925bb1","Type":"ContainerDied","Data":"210b771f8d3b3f4fcec054ca78d6b87dd048ab9c2c68c06633aa0f67e2ec5ebe"} Nov 02 22:29:32.188532 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.188502 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5961abd-9636-44b4-b650-4a2a65c4b603" path="/var/lib/kubelet/pods/d5961abd-9636-44b4-b650-4a2a65c4b603/volumes" Nov 02 22:29:32.296076 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.296051 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:32.367038 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.367012 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-xks-ca\") pod \"96fe7b38-8085-493d-82cb-650451925bb1\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " Nov 02 22:29:32.367188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.367131 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-openshift-ca\") pod \"96fe7b38-8085-493d-82cb-650451925bb1\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " Nov 02 22:29:32.367188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.367165 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fe7b38-8085-493d-82cb-650451925bb1-kube-api-access-v7mdr\" (UniqueName: \"kubernetes.io/projected/96fe7b38-8085-493d-82cb-650451925bb1-kube-api-access-v7mdr\") pod \"96fe7b38-8085-493d-82cb-650451925bb1\" (UID: \"96fe7b38-8085-493d-82cb-650451925bb1\") " Nov 02 22:29:32.367523 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.367502 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-openshift-ca" pod "96fe7b38-8085-493d-82cb-650451925bb1" (UID: "96fe7b38-8085-493d-82cb-650451925bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:32.369145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.369117 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-xks-ca" pod "96fe7b38-8085-493d-82cb-650451925bb1" (UID: "96fe7b38-8085-493d-82cb-650451925bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:32.369252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.369144 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fe7b38-8085-493d-82cb-650451925bb1-kube-api-access-v7mdr" pod "96fe7b38-8085-493d-82cb-650451925bb1" (UID: "96fe7b38-8085-493d-82cb-650451925bb1"). InnerVolumeSpecName "kube-api-access-v7mdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:32.468994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.468923 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:32.468994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.468950 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mdr\" (UniqueName: \"kubernetes.io/projected/96fe7b38-8085-493d-82cb-650451925bb1-kube-api-access-v7mdr\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:32.468994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:32.468960 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96fe7b38-8085-493d-82cb-650451925bb1-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:33.170352 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:33.170326 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" Nov 02 22:29:33.170518 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:33.170357 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z" event={"ID":"96fe7b38-8085-493d-82cb-650451925bb1","Type":"ContainerDied","Data":"17283b60aead2ca71761e2ac69482766da5abaf89a18b3549a46a55a805dbac9"} Nov 02 22:29:33.170518 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:33.170405 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17283b60aead2ca71761e2ac69482766da5abaf89a18b3549a46a55a805dbac9" Nov 02 22:29:39.153270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.153235 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt"] Nov 02 22:29:39.153831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.153702 4309 cpu_manager.go:414] "containerMap: removing container" podUID="96fe7b38-8085-493d-82cb-650451925bb1" containerName="revoke-keys" Nov 02 22:29:39.153831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.153715 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96fe7b38-8085-493d-82cb-650451925bb1" containerName="revoke-keys" Nov 02 22:29:39.153831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.153809 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="96fe7b38-8085-493d-82cb-650451925bb1" containerName="revoke-keys" Nov 02 22:29:39.157379 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.157358 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.165734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.162367 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:29:39.165734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.162810 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:29:39.165734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.162855 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:39.165734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.162905 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:29:39.167389 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.167355 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt"] Nov 02 22:29:39.239883 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.239849 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcfwz\" (UniqueName: \"kubernetes.io/projected/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-kube-api-access-lcfwz\") pod \"maas-api-revoke-sub-d3fba056-tl4pt\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.240043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.240027 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tl4pt\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.240089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.240052 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tl4pt\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.340792 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.340759 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tl4pt\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.341001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.340801 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tl4pt\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.341001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.340848 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcfwz\" (UniqueName: \"kubernetes.io/projected/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-kube-api-access-lcfwz\") pod \"maas-api-revoke-sub-d3fba056-tl4pt\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.341540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.341513 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-tl4pt\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.343207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.343185 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-tl4pt\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.349043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.349020 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcfwz\" (UniqueName: \"kubernetes.io/projected/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-kube-api-access-lcfwz\") pod \"maas-api-revoke-sub-d3fba056-tl4pt\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.478684 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.478578 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:39.754992 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:39.754961 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt"] Nov 02 22:29:39.756105 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:29:39.756079 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba14f814_d8ca_47ab_9c0f_d5dcf0417294.slice/crio-aafd781e63f3a0759fcfa6dfc2ce629039cd212ca1bcabe5db11a33edbe90a3a WatchSource:0}: Error finding container aafd781e63f3a0759fcfa6dfc2ce629039cd212ca1bcabe5db11a33edbe90a3a: Status 404 returned error can't find the container with id aafd781e63f3a0759fcfa6dfc2ce629039cd212ca1bcabe5db11a33edbe90a3a Nov 02 22:29:40.214430 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:40.214390 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" event={"ID":"ba14f814-d8ca-47ab-9c0f-d5dcf0417294","Type":"ContainerStarted","Data":"2610d98ad3267207cf41288fc903c15b8d6d3c451972dc2a5320d49b3d845a4a"} Nov 02 22:29:40.214973 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:40.214946 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" event={"ID":"ba14f814-d8ca-47ab-9c0f-d5dcf0417294","Type":"ContainerStarted","Data":"aafd781e63f3a0759fcfa6dfc2ce629039cd212ca1bcabe5db11a33edbe90a3a"} Nov 02 22:29:40.233865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:40.233811 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" podStartSLOduration=1.233792692 podStartE2EDuration="1.233792692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:29:40.229910917 +0000 UTC m=+1908.168024150" watchObservedRunningTime="2026-11-02 22:29:40.233792692 +0000 UTC m=+1908.171905912" Nov 02 22:29:41.249216 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:41.249173 4309 generic.go:356] "Generic (PLEG): container finished" podID="ba14f814-d8ca-47ab-9c0f-d5dcf0417294" containerID="2610d98ad3267207cf41288fc903c15b8d6d3c451972dc2a5320d49b3d845a4a" exitCode=0 Nov 02 22:29:41.249978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:41.249951 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" event={"ID":"ba14f814-d8ca-47ab-9c0f-d5dcf0417294","Type":"ContainerDied","Data":"2610d98ad3267207cf41288fc903c15b8d6d3c451972dc2a5320d49b3d845a4a"} Nov 02 22:29:41.680724 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:29:41.680661 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:29:42.785361 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.785318 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:42.891321 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.891281 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-openshift-ca\") pod \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " Nov 02 22:29:42.891530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.891408 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-kube-api-access-lcfwz\" (UniqueName: \"kubernetes.io/projected/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-kube-api-access-lcfwz\") pod \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " Nov 02 22:29:42.891530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.891482 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-xks-ca\") pod \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\" (UID: \"ba14f814-d8ca-47ab-9c0f-d5dcf0417294\") " Nov 02 22:29:42.892798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.892771 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-openshift-ca" pod "ba14f814-d8ca-47ab-9c0f-d5dcf0417294" (UID: "ba14f814-d8ca-47ab-9c0f-d5dcf0417294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:29:42.902408 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.902369 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-xks-ca" pod "ba14f814-d8ca-47ab-9c0f-d5dcf0417294" (UID: "ba14f814-d8ca-47ab-9c0f-d5dcf0417294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:29:42.902652 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.902621 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-kube-api-access-lcfwz" pod "ba14f814-d8ca-47ab-9c0f-d5dcf0417294" (UID: "ba14f814-d8ca-47ab-9c0f-d5dcf0417294"). InnerVolumeSpecName "kube-api-access-lcfwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:29:42.993253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.993165 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:42.993253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.993197 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcfwz\" (UniqueName: \"kubernetes.io/projected/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-kube-api-access-lcfwz\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:42.993253 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:42.993208 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba14f814-d8ca-47ab-9c0f-d5dcf0417294-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:29:43.343361 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:43.343327 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" event={"ID":"ba14f814-d8ca-47ab-9c0f-d5dcf0417294","Type":"ContainerDied","Data":"aafd781e63f3a0759fcfa6dfc2ce629039cd212ca1bcabe5db11a33edbe90a3a"} Nov 02 22:29:43.343549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:43.343366 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt" Nov 02 22:29:43.343549 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:43.343376 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aafd781e63f3a0759fcfa6dfc2ce629039cd212ca1bcabe5db11a33edbe90a3a" Nov 02 22:29:45.022066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:45.022024 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql"] Nov 02 22:29:45.024811 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:45.024784 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37acced6-2gbql"] Nov 02 22:29:46.188026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:46.187995 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af63f12-8f7b-4505-ae0f-99e851824cd6" path="/var/lib/kubelet/pods/7af63f12-8f7b-4505-ae0f-99e851824cd6/volumes" Nov 02 22:29:49.020354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:49.020317 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll"] Nov 02 22:29:49.022863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:49.022841 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b8200ed-blvll"] Nov 02 22:29:50.189328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:50.189293 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00258507-e531-4b1f-a3ad-fa55c532c503" path="/var/lib/kubelet/pods/00258507-e531-4b1f-a3ad-fa55c532c503/volumes" Nov 02 22:29:55.019567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:55.019528 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2"] Nov 02 22:29:55.022039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:55.022014 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a329dc5c-p42w2"] Nov 02 22:29:56.188804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:56.188767 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1c3a1b-62e3-462f-9349-16d2665af0c2" path="/var/lib/kubelet/pods/1d1c3a1b-62e3-462f-9349-16d2665af0c2/volumes" Nov 02 22:29:59.788534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.788496 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4"] Nov 02 22:29:59.789035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.789013 4309 cpu_manager.go:414] "containerMap: removing container" podUID="ba14f814-d8ca-47ab-9c0f-d5dcf0417294" containerName="revoke-keys" Nov 02 22:29:59.789035 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.789027 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba14f814-d8ca-47ab-9c0f-d5dcf0417294" containerName="revoke-keys" Nov 02 22:29:59.789135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.789122 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba14f814-d8ca-47ab-9c0f-d5dcf0417294" containerName="revoke-keys" Nov 02 22:29:59.792648 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.792631 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:29:59.796707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.796655 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:29:59.796707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.796698 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:29:59.796928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.796666 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:29:59.797153 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.797137 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:29:59.798991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.798968 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4"] Nov 02 22:29:59.905292 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.905262 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-vfnc4\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:29:59.905458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.905309 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrwdz\" (UniqueName: \"kubernetes.io/projected/288fb5ad-2361-44de-ba36-d4ff7a4b0799-kube-api-access-jrwdz\") pod \"maas-api-revoke-sub-ae594979-vfnc4\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:29:59.905458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:29:59.905406 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-vfnc4\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:00.006943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:00.006903 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-vfnc4\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:00.007124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:00.006968 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrwdz\" (UniqueName: \"kubernetes.io/projected/288fb5ad-2361-44de-ba36-d4ff7a4b0799-kube-api-access-jrwdz\") pod \"maas-api-revoke-sub-ae594979-vfnc4\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:00.007124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:00.007011 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-vfnc4\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:00.007740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:00.007708 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-vfnc4\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:00.009261 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:00.009243 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-vfnc4\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:00.015479 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:00.015453 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrwdz\" (UniqueName: \"kubernetes.io/projected/288fb5ad-2361-44de-ba36-d4ff7a4b0799-kube-api-access-jrwdz\") pod \"maas-api-revoke-sub-ae594979-vfnc4\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:00.108257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:00.108215 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:00.386156 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:00.386126 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4"] Nov 02 22:30:00.387010 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:30:00.386981 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288fb5ad_2361_44de_ba36_d4ff7a4b0799.slice/crio-248c01e1ffde51a3b4953c23017e323ec01a1fcf5593aaceba527aa1b18dd3ec WatchSource:0}: Error finding container 248c01e1ffde51a3b4953c23017e323ec01a1fcf5593aaceba527aa1b18dd3ec: Status 404 returned error can't find the container with id 248c01e1ffde51a3b4953c23017e323ec01a1fcf5593aaceba527aa1b18dd3ec Nov 02 22:30:00.477868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:00.477835 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" event={"ID":"288fb5ad-2361-44de-ba36-d4ff7a4b0799","Type":"ContainerStarted","Data":"248c01e1ffde51a3b4953c23017e323ec01a1fcf5593aaceba527aa1b18dd3ec"} Nov 02 22:30:01.486026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:01.485937 4309 generic.go:356] "Generic (PLEG): container finished" podID="288fb5ad-2361-44de-ba36-d4ff7a4b0799" containerID="89108629198a05431cc20c8b1806d8cf75898f1fbc3a37726118b3a1bba2d229" exitCode=0 Nov 02 22:30:01.486412 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:01.486023 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" event={"ID":"288fb5ad-2361-44de-ba36-d4ff7a4b0799","Type":"ContainerDied","Data":"89108629198a05431cc20c8b1806d8cf75898f1fbc3a37726118b3a1bba2d229"} Nov 02 22:30:02.632587 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.632558 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:02.634579 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.634556 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-openshift-ca\") pod \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " Nov 02 22:30:02.634711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.634641 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288fb5ad-2361-44de-ba36-d4ff7a4b0799-kube-api-access-jrwdz\" (UniqueName: \"kubernetes.io/projected/288fb5ad-2361-44de-ba36-d4ff7a4b0799-kube-api-access-jrwdz\") pod \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " Nov 02 22:30:02.634782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.634758 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-xks-ca\") pod \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\" (UID: \"288fb5ad-2361-44de-ba36-d4ff7a4b0799\") " Nov 02 22:30:02.634950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.634923 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-openshift-ca" pod "288fb5ad-2361-44de-ba36-d4ff7a4b0799" (UID: "288fb5ad-2361-44de-ba36-d4ff7a4b0799"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:02.635075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.635060 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:02.636821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.636800 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-xks-ca" pod "288fb5ad-2361-44de-ba36-d4ff7a4b0799" (UID: "288fb5ad-2361-44de-ba36-d4ff7a4b0799"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:02.636907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.636836 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288fb5ad-2361-44de-ba36-d4ff7a4b0799-kube-api-access-jrwdz" pod "288fb5ad-2361-44de-ba36-d4ff7a4b0799" (UID: "288fb5ad-2361-44de-ba36-d4ff7a4b0799"). InnerVolumeSpecName "kube-api-access-jrwdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:02.735907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.735871 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrwdz\" (UniqueName: \"kubernetes.io/projected/288fb5ad-2361-44de-ba36-d4ff7a4b0799-kube-api-access-jrwdz\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:02.735907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:02.735904 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/288fb5ad-2361-44de-ba36-d4ff7a4b0799-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:03.020727 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:03.020667 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf"] Nov 02 22:30:03.023217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:03.023192 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be757162-qn8gf"] Nov 02 22:30:03.499409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:03.499376 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" Nov 02 22:30:03.499605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:03.499429 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4" event={"ID":"288fb5ad-2361-44de-ba36-d4ff7a4b0799","Type":"ContainerDied","Data":"248c01e1ffde51a3b4953c23017e323ec01a1fcf5593aaceba527aa1b18dd3ec"} Nov 02 22:30:03.499605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:03.499477 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="248c01e1ffde51a3b4953c23017e323ec01a1fcf5593aaceba527aa1b18dd3ec" Nov 02 22:30:04.187966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:04.187932 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23bf9d1-a705-467c-b7b2-d955c986b4bf" path="/var/lib/kubelet/pods/f23bf9d1-a705-467c-b7b2-d955c986b4bf/volumes" Nov 02 22:30:09.022444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:09.022353 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r"] Nov 02 22:30:09.025043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:09.025014 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-982b8836-tkc2r"] Nov 02 22:30:10.187609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:10.187570 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a11a862-1ca2-43d5-9847-46e7113b9908" path="/var/lib/kubelet/pods/3a11a862-1ca2-43d5-9847-46e7113b9908/volumes" Nov 02 22:30:11.595898 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:30:11.595857 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:30:17.957607 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.957571 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb"] Nov 02 22:30:17.958033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.958020 4309 cpu_manager.go:414] "containerMap: removing container" podUID="288fb5ad-2361-44de-ba36-d4ff7a4b0799" containerName="revoke-keys" Nov 02 22:30:17.958079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.958034 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288fb5ad-2361-44de-ba36-d4ff7a4b0799" containerName="revoke-keys" Nov 02 22:30:17.958148 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.958139 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="288fb5ad-2361-44de-ba36-d4ff7a4b0799" containerName="revoke-keys" Nov 02 22:30:17.961241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.961222 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:17.965733 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.965708 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:30:17.966387 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.966359 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:30:17.966520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.966392 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:30:17.966520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.966396 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:30:17.969590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:17.969565 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb"] Nov 02 22:30:18.007184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.007144 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dtnlb\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.007360 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.007193 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvknr\" (UniqueName: \"kubernetes.io/projected/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-kube-api-access-zvknr\") pod \"maas-api-revoke-sub-a4aef7d7-dtnlb\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.007410 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.007356 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dtnlb\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.108517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.108466 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dtnlb\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.108517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.108511 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvknr\" (UniqueName: \"kubernetes.io/projected/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-kube-api-access-zvknr\") pod \"maas-api-revoke-sub-a4aef7d7-dtnlb\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.108834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.108580 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dtnlb\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.109264 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.109244 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dtnlb\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.110796 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.110776 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dtnlb\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.116085 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.116062 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvknr\" (UniqueName: \"kubernetes.io/projected/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-kube-api-access-zvknr\") pod \"maas-api-revoke-sub-a4aef7d7-dtnlb\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.276436 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.276354 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:18.553507 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.553485 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb"] Nov 02 22:30:18.555019 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:30:18.554988 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a20d16_10da_4b9b_8fa5_acdcc1cc7082.slice/crio-c35f476b30e51bfda608966bd360c2754ca1d1f351b245356fcff550c6ad7713 WatchSource:0}: Error finding container c35f476b30e51bfda608966bd360c2754ca1d1f351b245356fcff550c6ad7713: Status 404 returned error can't find the container with id c35f476b30e51bfda608966bd360c2754ca1d1f351b245356fcff550c6ad7713 Nov 02 22:30:18.610090 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:18.610053 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" event={"ID":"22a20d16-10da-4b9b-8fa5-acdcc1cc7082","Type":"ContainerStarted","Data":"c35f476b30e51bfda608966bd360c2754ca1d1f351b245356fcff550c6ad7713"} Nov 02 22:30:19.617667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:19.617632 4309 generic.go:356] "Generic (PLEG): container finished" podID="22a20d16-10da-4b9b-8fa5-acdcc1cc7082" containerID="afd88bb4960e9de7098c2b1144d298c881a2ec4352c40e50577109a7f65bb3e9" exitCode=0 Nov 02 22:30:19.618072 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:19.617717 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" event={"ID":"22a20d16-10da-4b9b-8fa5-acdcc1cc7082","Type":"ContainerDied","Data":"afd88bb4960e9de7098c2b1144d298c881a2ec4352c40e50577109a7f65bb3e9"} Nov 02 22:30:20.759409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.759385 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:20.836904 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.836865 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-openshift-ca\") pod \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " Nov 02 22:30:20.837107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.837012 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-kube-api-access-zvknr\" (UniqueName: \"kubernetes.io/projected/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-kube-api-access-zvknr\") pod \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " Nov 02 22:30:20.837107 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.837034 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-xks-ca\") pod \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\" (UID: \"22a20d16-10da-4b9b-8fa5-acdcc1cc7082\") " Nov 02 22:30:20.837318 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.837291 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-openshift-ca" pod "22a20d16-10da-4b9b-8fa5-acdcc1cc7082" (UID: "22a20d16-10da-4b9b-8fa5-acdcc1cc7082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:20.839080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.839055 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-kube-api-access-zvknr" pod "22a20d16-10da-4b9b-8fa5-acdcc1cc7082" (UID: "22a20d16-10da-4b9b-8fa5-acdcc1cc7082"). InnerVolumeSpecName "kube-api-access-zvknr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:20.839185 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.839111 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-xks-ca" pod "22a20d16-10da-4b9b-8fa5-acdcc1cc7082" (UID: "22a20d16-10da-4b9b-8fa5-acdcc1cc7082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:20.938582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.938495 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvknr\" (UniqueName: \"kubernetes.io/projected/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-kube-api-access-zvknr\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:20.938582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.938528 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:20.938582 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:20.938543 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22a20d16-10da-4b9b-8fa5-acdcc1cc7082-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:21.631422 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:21.631393 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" Nov 02 22:30:21.631630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:21.631392 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb" event={"ID":"22a20d16-10da-4b9b-8fa5-acdcc1cc7082","Type":"ContainerDied","Data":"c35f476b30e51bfda608966bd360c2754ca1d1f351b245356fcff550c6ad7713"} Nov 02 22:30:21.631630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:21.631540 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c35f476b30e51bfda608966bd360c2754ca1d1f351b245356fcff550c6ad7713" Nov 02 22:30:22.019514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:22.019427 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg"] Nov 02 22:30:22.022275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:22.022247 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a82826b6-szkhg"] Nov 02 22:30:22.188991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:22.188959 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edace62b-0f08-47f4-b6d4-83b0952e8242" path="/var/lib/kubelet/pods/edace62b-0f08-47f4-b6d4-83b0952e8242/volumes" Nov 02 22:30:36.581376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.581343 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld"] Nov 02 22:30:36.581884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.581819 4309 cpu_manager.go:414] "containerMap: removing container" podUID="22a20d16-10da-4b9b-8fa5-acdcc1cc7082" containerName="revoke-keys" Nov 02 22:30:36.581884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.581835 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a20d16-10da-4b9b-8fa5-acdcc1cc7082" containerName="revoke-keys" Nov 02 22:30:36.581967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.581914 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a20d16-10da-4b9b-8fa5-acdcc1cc7082" containerName="revoke-keys" Nov 02 22:30:36.586229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.586213 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.592720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.590297 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:30:36.592720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.590559 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:30:36.592720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.591413 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:30:36.592720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.591579 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:30:36.593079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.592769 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld"] Nov 02 22:30:36.710789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.710760 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n522s\" (UniqueName: \"kubernetes.io/projected/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-kube-api-access-n522s\") pod \"maas-api-revoke-sub-79805ca9-l6jld\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.710956 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.710807 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-l6jld\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.711008 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.710976 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-l6jld\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.811687 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.811653 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-l6jld\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.811889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.811757 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-l6jld\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.811889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.811831 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n522s\" (UniqueName: \"kubernetes.io/projected/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-kube-api-access-n522s\") pod \"maas-api-revoke-sub-79805ca9-l6jld\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.812477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.812456 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-l6jld\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.814041 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.814018 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-l6jld\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.819361 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.819344 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n522s\" (UniqueName: \"kubernetes.io/projected/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-kube-api-access-n522s\") pod \"maas-api-revoke-sub-79805ca9-l6jld\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:36.904041 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:36.903961 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:37.180006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:37.179976 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld"] Nov 02 22:30:37.180851 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:30:37.180822 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d0782c_f5f0_4a2f_a16f_03edcb53bcb8.slice/crio-9587c9e83771589760a4b92ef98ad8d9281e4f2e51adc833ce7ad1899eba2eb4 WatchSource:0}: Error finding container 9587c9e83771589760a4b92ef98ad8d9281e4f2e51adc833ce7ad1899eba2eb4: Status 404 returned error can't find the container with id 9587c9e83771589760a4b92ef98ad8d9281e4f2e51adc833ce7ad1899eba2eb4 Nov 02 22:30:37.775356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:37.775268 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" event={"ID":"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8","Type":"ContainerStarted","Data":"c71d96b0b9f8a99716e0e6550a8b50f4026c7e1fcd621eb15261651459682a4a"} Nov 02 22:30:37.775356 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:37.775325 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" event={"ID":"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8","Type":"ContainerStarted","Data":"9587c9e83771589760a4b92ef98ad8d9281e4f2e51adc833ce7ad1899eba2eb4"} Nov 02 22:30:37.793965 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:37.793911 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" podStartSLOduration=1.7938970699999999 podStartE2EDuration="1.79389707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:30:37.790886907 +0000 UTC m=+1965.729000126" watchObservedRunningTime="2026-11-02 22:30:37.79389707 +0000 UTC m=+1965.732010303" Nov 02 22:30:38.783798 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:38.783766 4309 generic.go:356] "Generic (PLEG): container finished" podID="f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8" containerID="c71d96b0b9f8a99716e0e6550a8b50f4026c7e1fcd621eb15261651459682a4a" exitCode=0 Nov 02 22:30:38.784258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:38.783856 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" event={"ID":"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8","Type":"ContainerDied","Data":"c71d96b0b9f8a99716e0e6550a8b50f4026c7e1fcd621eb15261651459682a4a"} Nov 02 22:30:39.927144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:39.927114 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:40.045376 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.045279 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-openshift-ca\") pod \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " Nov 02 22:30:40.045576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.045416 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-kube-api-access-n522s\" (UniqueName: \"kubernetes.io/projected/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-kube-api-access-n522s\") pod \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " Nov 02 22:30:40.045576 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.045473 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-xks-ca\") pod \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\" (UID: \"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8\") " Nov 02 22:30:40.045778 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.045747 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-openshift-ca" pod "f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8" (UID: "f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:40.045892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.045878 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.047588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.047570 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-kube-api-access-n522s" pod "f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8" (UID: "f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8"). InnerVolumeSpecName "kube-api-access-n522s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:40.047588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.047573 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-xks-ca" pod "f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8" (UID: "f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:40.147353 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.147315 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n522s\" (UniqueName: \"kubernetes.io/projected/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-kube-api-access-n522s\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.147353 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.147349 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:40.798591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.798553 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" Nov 02 22:30:40.798784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.798553 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld" event={"ID":"f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8","Type":"ContainerDied","Data":"9587c9e83771589760a4b92ef98ad8d9281e4f2e51adc833ce7ad1899eba2eb4"} Nov 02 22:30:40.798784 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:40.798739 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9587c9e83771589760a4b92ef98ad8d9281e4f2e51adc833ce7ad1899eba2eb4" Nov 02 22:30:41.595565 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:30:41.595529 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:30:54.311824 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.311791 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn"] Nov 02 22:30:54.312258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.312243 4309 cpu_manager.go:414] "containerMap: removing container" podUID="f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8" containerName="revoke-keys" Nov 02 22:30:54.312304 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.312259 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8" containerName="revoke-keys" Nov 02 22:30:54.312349 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.312340 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8" containerName="revoke-keys" Nov 02 22:30:54.315745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.315729 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.319786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.319663 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nb7fn\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.319786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.319737 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:30:54.319786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.319754 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:30:54.319786 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.319739 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:30:54.320016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.319946 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmq5k\" (UniqueName: \"kubernetes.io/projected/675d7445-63dc-4369-99c0-dbc3382265b9-kube-api-access-rmq5k\") pod \"maas-api-revoke-sub-0b7e8edb-nb7fn\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.320016 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.319995 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nb7fn\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.320220 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.320199 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:30:54.322596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.322574 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn"] Nov 02 22:30:54.421427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.421394 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmq5k\" (UniqueName: \"kubernetes.io/projected/675d7445-63dc-4369-99c0-dbc3382265b9-kube-api-access-rmq5k\") pod \"maas-api-revoke-sub-0b7e8edb-nb7fn\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.421631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.421432 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nb7fn\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.421631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.421466 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nb7fn\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.422166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.422137 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nb7fn\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.423802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.423783 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-nb7fn\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.429007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.428981 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmq5k\" (UniqueName: \"kubernetes.io/projected/675d7445-63dc-4369-99c0-dbc3382265b9-kube-api-access-rmq5k\") pod \"maas-api-revoke-sub-0b7e8edb-nb7fn\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.632014 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.631926 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:54.907304 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:54.907278 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn"] Nov 02 22:30:54.908035 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:30:54.908011 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675d7445_63dc_4369_99c0_dbc3382265b9.slice/crio-cc9d4899c9a63114457d45caba82d434aac37832211bf3f5863775ecc29b1b34 WatchSource:0}: Error finding container cc9d4899c9a63114457d45caba82d434aac37832211bf3f5863775ecc29b1b34: Status 404 returned error can't find the container with id cc9d4899c9a63114457d45caba82d434aac37832211bf3f5863775ecc29b1b34 Nov 02 22:30:55.919787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:55.919747 4309 generic.go:356] "Generic (PLEG): container finished" podID="675d7445-63dc-4369-99c0-dbc3382265b9" containerID="c17f4a0ab97ccd5d432913df4de2623ecbd3a0be1890c55a36e7024004d2ae40" exitCode=0 Nov 02 22:30:55.920170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:55.919794 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" event={"ID":"675d7445-63dc-4369-99c0-dbc3382265b9","Type":"ContainerDied","Data":"c17f4a0ab97ccd5d432913df4de2623ecbd3a0be1890c55a36e7024004d2ae40"} Nov 02 22:30:55.920170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:55.919840 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" event={"ID":"675d7445-63dc-4369-99c0-dbc3382265b9","Type":"ContainerStarted","Data":"cc9d4899c9a63114457d45caba82d434aac37832211bf3f5863775ecc29b1b34"} Nov 02 22:30:57.064357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.064331 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:57.148502 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.148473 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-xks-ca\") pod \"675d7445-63dc-4369-99c0-dbc3382265b9\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " Nov 02 22:30:57.148666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.148507 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-openshift-ca\") pod \"675d7445-63dc-4369-99c0-dbc3382265b9\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " Nov 02 22:30:57.148666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.148556 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675d7445-63dc-4369-99c0-dbc3382265b9-kube-api-access-rmq5k\" (UniqueName: \"kubernetes.io/projected/675d7445-63dc-4369-99c0-dbc3382265b9-kube-api-access-rmq5k\") pod \"675d7445-63dc-4369-99c0-dbc3382265b9\" (UID: \"675d7445-63dc-4369-99c0-dbc3382265b9\") " Nov 02 22:30:57.149028 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.148997 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-openshift-ca" pod "675d7445-63dc-4369-99c0-dbc3382265b9" (UID: "675d7445-63dc-4369-99c0-dbc3382265b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:30:57.150663 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.150638 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-xks-ca" pod "675d7445-63dc-4369-99c0-dbc3382265b9" (UID: "675d7445-63dc-4369-99c0-dbc3382265b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:30:57.150803 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.150704 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675d7445-63dc-4369-99c0-dbc3382265b9-kube-api-access-rmq5k" pod "675d7445-63dc-4369-99c0-dbc3382265b9" (UID: "675d7445-63dc-4369-99c0-dbc3382265b9"). InnerVolumeSpecName "kube-api-access-rmq5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:30:57.250098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.250010 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:57.250098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.250038 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/675d7445-63dc-4369-99c0-dbc3382265b9-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:57.250098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.250048 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmq5k\" (UniqueName: \"kubernetes.io/projected/675d7445-63dc-4369-99c0-dbc3382265b9-kube-api-access-rmq5k\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:30:57.934088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.934049 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" event={"ID":"675d7445-63dc-4369-99c0-dbc3382265b9","Type":"ContainerDied","Data":"cc9d4899c9a63114457d45caba82d434aac37832211bf3f5863775ecc29b1b34"} Nov 02 22:30:57.934088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.934078 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn" Nov 02 22:30:57.934313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:30:57.934110 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc9d4899c9a63114457d45caba82d434aac37832211bf3f5863775ecc29b1b34" Nov 02 22:31:00.524870 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:00.524837 4309 scope.go:122] "RemoveContainer" containerID="ee1a12a2065550297c6a249c6b8163b80062b2816a87e0014c7a5ed92db04acf" Nov 02 22:31:00.535069 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:00.535046 4309 scope.go:122] "RemoveContainer" containerID="0811a8941f0fe9a56123d51ba59bdc97071dd05e5eced04b2d4052f110bafc3d" Nov 02 22:31:00.543509 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:00.543491 4309 scope.go:122] "RemoveContainer" containerID="c54a53de1a7e3d2ed40166ae9b8abdb683aa38a16ba091a452f840131e92b44a" Nov 02 22:31:00.551738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:00.551723 4309 scope.go:122] "RemoveContainer" containerID="c0eb787e009d05007abf806b7a6628d562cbd708b17957198ad914d3283612ce" Nov 02 22:31:00.559814 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:00.559795 4309 scope.go:122] "RemoveContainer" containerID="5af98d36239a5d3a3f7a5042250843b99f28b6460e0c75c04ae8e1246cb1e441" Nov 02 22:31:00.567995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:00.567974 4309 scope.go:122] "RemoveContainer" containerID="32f7f31c9569236e20444e337b318fd7442aa761ebcb3c86e526a972c5b47b35" Nov 02 22:31:00.577310 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:00.577294 4309 scope.go:122] "RemoveContainer" containerID="ff4e27bd8ac0d5ed2c52a6dd497a8e49a7b8330a7f30f69799899516a3eb937c" Nov 02 22:31:05.004540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.004507 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4"] Nov 02 22:31:05.005019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.004995 4309 cpu_manager.go:414] "containerMap: removing container" podUID="675d7445-63dc-4369-99c0-dbc3382265b9" containerName="revoke-keys" Nov 02 22:31:05.005069 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.005019 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675d7445-63dc-4369-99c0-dbc3382265b9" containerName="revoke-keys" Nov 02 22:31:05.005106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.005096 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="675d7445-63dc-4369-99c0-dbc3382265b9" containerName="revoke-keys" Nov 02 22:31:05.010691 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.010650 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.017112 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.016129 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4"] Nov 02 22:31:05.018704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.017332 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:31:05.018704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.017351 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:31:05.018704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.018269 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:31:05.018704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.018308 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:31:05.132954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.132918 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c60464-1632-47f4-916d-4291869faa14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6rbs4\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.133144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.133075 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c60464-1632-47f4-916d-4291869faa14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6rbs4\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.133144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.133124 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q96xd\" (UniqueName: \"kubernetes.io/projected/85c60464-1632-47f4-916d-4291869faa14-kube-api-access-q96xd\") pod \"maas-api-revoke-sub-3d4156f1-6rbs4\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.234476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.234442 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c60464-1632-47f4-916d-4291869faa14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6rbs4\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.234476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.234490 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q96xd\" (UniqueName: \"kubernetes.io/projected/85c60464-1632-47f4-916d-4291869faa14-kube-api-access-q96xd\") pod \"maas-api-revoke-sub-3d4156f1-6rbs4\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.234751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.234542 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c60464-1632-47f4-916d-4291869faa14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6rbs4\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.235147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.235115 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c60464-1632-47f4-916d-4291869faa14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6rbs4\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.237066 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.237040 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c60464-1632-47f4-916d-4291869faa14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6rbs4\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.242326 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.242302 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q96xd\" (UniqueName: \"kubernetes.io/projected/85c60464-1632-47f4-916d-4291869faa14-kube-api-access-q96xd\") pod \"maas-api-revoke-sub-3d4156f1-6rbs4\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.330471 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.330431 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:05.615342 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.615263 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4"] Nov 02 22:31:05.623846 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:31:05.623814 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c60464_1632_47f4_916d_4291869faa14.slice/crio-884bb0cd5bbdeec2db7bede6b055139477fa5ad79999563a8b57f92a6df03683 WatchSource:0}: Error finding container 884bb0cd5bbdeec2db7bede6b055139477fa5ad79999563a8b57f92a6df03683: Status 404 returned error can't find the container with id 884bb0cd5bbdeec2db7bede6b055139477fa5ad79999563a8b57f92a6df03683 Nov 02 22:31:05.985150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.985109 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" event={"ID":"85c60464-1632-47f4-916d-4291869faa14","Type":"ContainerStarted","Data":"006988607ef8a3e1365f2b8e0e2e8d1df68d61d443842ceaba409a68ace2edf2"} Nov 02 22:31:05.985334 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:05.985173 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" event={"ID":"85c60464-1632-47f4-916d-4291869faa14","Type":"ContainerStarted","Data":"884bb0cd5bbdeec2db7bede6b055139477fa5ad79999563a8b57f92a6df03683"} Nov 02 22:31:06.003203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:06.003151 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" podStartSLOduration=2.003137383 podStartE2EDuration="2.003137383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:05.999884497 +0000 UTC m=+1993.937997717" watchObservedRunningTime="2026-11-02 22:31:06.003137383 +0000 UTC m=+1993.941250614" Nov 02 22:31:06.993026 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:06.992992 4309 generic.go:356] "Generic (PLEG): container finished" podID="85c60464-1632-47f4-916d-4291869faa14" containerID="006988607ef8a3e1365f2b8e0e2e8d1df68d61d443842ceaba409a68ace2edf2" exitCode=0 Nov 02 22:31:06.993478 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:06.993078 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" event={"ID":"85c60464-1632-47f4-916d-4291869faa14","Type":"ContainerDied","Data":"006988607ef8a3e1365f2b8e0e2e8d1df68d61d443842ceaba409a68ace2edf2"} Nov 02 22:31:08.142849 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.142819 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:08.264958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.264884 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85c60464-1632-47f4-916d-4291869faa14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c60464-1632-47f4-916d-4291869faa14-maas-api-xks-ca\") pod \"85c60464-1632-47f4-916d-4291869faa14\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " Nov 02 22:31:08.264958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.264954 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c60464-1632-47f4-916d-4291869faa14-kube-api-access-q96xd\" (UniqueName: \"kubernetes.io/projected/85c60464-1632-47f4-916d-4291869faa14-kube-api-access-q96xd\") pod \"85c60464-1632-47f4-916d-4291869faa14\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " Nov 02 22:31:08.265179 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.265061 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c60464-1632-47f4-916d-4291869faa14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c60464-1632-47f4-916d-4291869faa14-maas-api-openshift-ca\") pod \"85c60464-1632-47f4-916d-4291869faa14\" (UID: \"85c60464-1632-47f4-916d-4291869faa14\") " Nov 02 22:31:08.265443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.265416 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c60464-1632-47f4-916d-4291869faa14-maas-api-openshift-ca" pod "85c60464-1632-47f4-916d-4291869faa14" (UID: "85c60464-1632-47f4-916d-4291869faa14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:08.266987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.266966 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c60464-1632-47f4-916d-4291869faa14-maas-api-xks-ca" pod "85c60464-1632-47f4-916d-4291869faa14" (UID: "85c60464-1632-47f4-916d-4291869faa14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:08.267077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.266996 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c60464-1632-47f4-916d-4291869faa14-kube-api-access-q96xd" pod "85c60464-1632-47f4-916d-4291869faa14" (UID: "85c60464-1632-47f4-916d-4291869faa14"). InnerVolumeSpecName "kube-api-access-q96xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:08.366133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.366106 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c60464-1632-47f4-916d-4291869faa14-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:08.366133 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.366131 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c60464-1632-47f4-916d-4291869faa14-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:08.366308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:08.366141 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q96xd\" (UniqueName: \"kubernetes.io/projected/85c60464-1632-47f4-916d-4291869faa14-kube-api-access-q96xd\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:09.008309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:09.008276 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" event={"ID":"85c60464-1632-47f4-916d-4291869faa14","Type":"ContainerDied","Data":"884bb0cd5bbdeec2db7bede6b055139477fa5ad79999563a8b57f92a6df03683"} Nov 02 22:31:09.008309 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:09.008303 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4" Nov 02 22:31:09.008536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:09.008324 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="884bb0cd5bbdeec2db7bede6b055139477fa5ad79999563a8b57f92a6df03683" Nov 02 22:31:11.595622 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:31:11.595586 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:31:15.022432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:15.022391 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf"] Nov 02 22:31:15.027116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:15.027082 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w"] Nov 02 22:31:15.029574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:15.029553 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfdd38af-47snf"] Nov 02 22:31:15.032130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:15.032103 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d87d388-p852w"] Nov 02 22:31:16.187293 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:16.187259 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8" path="/var/lib/kubelet/pods/0f13b1b4-ef53-45cb-bbf4-a7c9b9e42bf8/volumes" Nov 02 22:31:16.187707 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:16.187662 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b702b3-84ec-462b-b7f7-c406b9a162de" path="/var/lib/kubelet/pods/c3b702b3-84ec-462b-b7f7-c406b9a162de/volumes" Nov 02 22:31:22.023849 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:22.023797 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg"] Nov 02 22:31:22.028184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:22.028155 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8"] Nov 02 22:31:22.031789 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:22.031768 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-755fa9e9-tfjsg"] Nov 02 22:31:22.034343 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:22.034326 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84c3190-pttf8"] Nov 02 22:31:22.188794 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:22.188762 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a108f3-2eba-48e1-b96b-a6022d20c72c" path="/var/lib/kubelet/pods/08a108f3-2eba-48e1-b96b-a6022d20c72c/volumes" Nov 02 22:31:22.189207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:22.189193 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bcf78e8-6820-435c-9b82-6d29a9dab312" path="/var/lib/kubelet/pods/1bcf78e8-6820-435c-9b82-6d29a9dab312/volumes" Nov 02 22:31:23.025887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.025852 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4"] Nov 02 22:31:23.029448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.029418 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6rbs4"] Nov 02 22:31:23.047477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.047447 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j"] Nov 02 22:31:23.047947 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.047933 4309 cpu_manager.go:414] "containerMap: removing container" podUID="85c60464-1632-47f4-916d-4291869faa14" containerName="revoke-keys" Nov 02 22:31:23.048021 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.047948 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c60464-1632-47f4-916d-4291869faa14" containerName="revoke-keys" Nov 02 22:31:23.048060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.048024 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c60464-1632-47f4-916d-4291869faa14" containerName="revoke-keys" Nov 02 22:31:23.051443 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.051429 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.055242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.055220 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:31:23.055446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.055431 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:31:23.057114 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.057096 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j"] Nov 02 22:31:23.083395 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.083367 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:31:23.083573 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.083421 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:31:23.136959 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.136927 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fnl7j\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.137137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.137059 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fnl7j\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.137137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.137116 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7t6k\" (UniqueName: \"kubernetes.io/projected/5ccdc497-94b0-4fe4-951c-84616411c6e9-kube-api-access-c7t6k\") pod \"maas-api-revoke-sub-3d4156f1-fnl7j\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.238530 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.238492 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fnl7j\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.238751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.238552 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7t6k\" (UniqueName: \"kubernetes.io/projected/5ccdc497-94b0-4fe4-951c-84616411c6e9-kube-api-access-c7t6k\") pod \"maas-api-revoke-sub-3d4156f1-fnl7j\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.238751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.238594 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fnl7j\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.239277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.239254 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fnl7j\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.240919 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.240897 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fnl7j\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.245980 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.245955 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7t6k\" (UniqueName: \"kubernetes.io/projected/5ccdc497-94b0-4fe4-951c-84616411c6e9-kube-api-access-c7t6k\") pod \"maas-api-revoke-sub-3d4156f1-fnl7j\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.367332 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.367297 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:23.640200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:23.640169 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j"] Nov 02 22:31:23.641334 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:31:23.641306 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccdc497_94b0_4fe4_951c_84616411c6e9.slice/crio-5492f24c471d55405ad933cc752c512a297920e0170ee2f170c3e5c4101c05d2 WatchSource:0}: Error finding container 5492f24c471d55405ad933cc752c512a297920e0170ee2f170c3e5c4101c05d2: Status 404 returned error can't find the container with id 5492f24c471d55405ad933cc752c512a297920e0170ee2f170c3e5c4101c05d2 Nov 02 22:31:24.133013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:24.132975 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" event={"ID":"5ccdc497-94b0-4fe4-951c-84616411c6e9","Type":"ContainerStarted","Data":"306c025135f9161321729e5337e64d67b0f12b4a820502dee8ba4f7349302dd8"} Nov 02 22:31:24.133411 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:24.133032 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" event={"ID":"5ccdc497-94b0-4fe4-951c-84616411c6e9","Type":"ContainerStarted","Data":"5492f24c471d55405ad933cc752c512a297920e0170ee2f170c3e5c4101c05d2"} Nov 02 22:31:24.150427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:24.150338 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" podStartSLOduration=1.150323465 podStartE2EDuration="1.150323465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:24.147150783 +0000 UTC m=+2012.085264001" watchObservedRunningTime="2026-11-02 22:31:24.150323465 +0000 UTC m=+2012.088436682" Nov 02 22:31:24.188467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:24.188439 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c60464-1632-47f4-916d-4291869faa14" path="/var/lib/kubelet/pods/85c60464-1632-47f4-916d-4291869faa14/volumes" Nov 02 22:31:25.140898 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:25.140867 4309 generic.go:356] "Generic (PLEG): container finished" podID="5ccdc497-94b0-4fe4-951c-84616411c6e9" containerID="306c025135f9161321729e5337e64d67b0f12b4a820502dee8ba4f7349302dd8" exitCode=0 Nov 02 22:31:25.141287 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:25.140955 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" event={"ID":"5ccdc497-94b0-4fe4-951c-84616411c6e9","Type":"ContainerDied","Data":"306c025135f9161321729e5337e64d67b0f12b4a820502dee8ba4f7349302dd8"} Nov 02 22:31:26.022606 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.022572 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf"] Nov 02 22:31:26.025564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.025527 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22aa40e9-mgqdf"] Nov 02 22:31:26.027833 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.027806 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz"] Nov 02 22:31:26.030147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.030122 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c821917-pwgmz"] Nov 02 22:31:26.187830 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.187793 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896a2b60-040c-44f1-b84c-d6fac267b653" path="/var/lib/kubelet/pods/896a2b60-040c-44f1-b84c-d6fac267b653/volumes" Nov 02 22:31:26.188294 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.188193 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a04793-1d76-4d00-9f00-7d7c48d6a3ac" path="/var/lib/kubelet/pods/92a04793-1d76-4d00-9f00-7d7c48d6a3ac/volumes" Nov 02 22:31:26.284552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.284496 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:26.370983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.370955 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ccdc497-94b0-4fe4-951c-84616411c6e9-kube-api-access-c7t6k\" (UniqueName: \"kubernetes.io/projected/5ccdc497-94b0-4fe4-951c-84616411c6e9-kube-api-access-c7t6k\") pod \"5ccdc497-94b0-4fe4-951c-84616411c6e9\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " Nov 02 22:31:26.371161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.371060 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-openshift-ca\") pod \"5ccdc497-94b0-4fe4-951c-84616411c6e9\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " Nov 02 22:31:26.371161 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.371134 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-xks-ca\") pod \"5ccdc497-94b0-4fe4-951c-84616411c6e9\" (UID: \"5ccdc497-94b0-4fe4-951c-84616411c6e9\") " Nov 02 22:31:26.371489 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.371451 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-openshift-ca" pod "5ccdc497-94b0-4fe4-951c-84616411c6e9" (UID: "5ccdc497-94b0-4fe4-951c-84616411c6e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:26.373082 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.373055 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ccdc497-94b0-4fe4-951c-84616411c6e9-kube-api-access-c7t6k" pod "5ccdc497-94b0-4fe4-951c-84616411c6e9" (UID: "5ccdc497-94b0-4fe4-951c-84616411c6e9"). InnerVolumeSpecName "kube-api-access-c7t6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:26.373188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.373156 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-xks-ca" pod "5ccdc497-94b0-4fe4-951c-84616411c6e9" (UID: "5ccdc497-94b0-4fe4-951c-84616411c6e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:26.472298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.472266 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:26.472298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.472295 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ccdc497-94b0-4fe4-951c-84616411c6e9-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:26.472298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:26.472305 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7t6k\" (UniqueName: \"kubernetes.io/projected/5ccdc497-94b0-4fe4-951c-84616411c6e9-kube-api-access-c7t6k\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:27.155865 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:27.155837 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" Nov 02 22:31:27.156093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:27.155836 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j" event={"ID":"5ccdc497-94b0-4fe4-951c-84616411c6e9","Type":"ContainerDied","Data":"5492f24c471d55405ad933cc752c512a297920e0170ee2f170c3e5c4101c05d2"} Nov 02 22:31:27.156093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:27.155952 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5492f24c471d55405ad933cc752c512a297920e0170ee2f170c3e5c4101c05d2" Nov 02 22:31:41.610541 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:31:41.610488 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:31:45.243671 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.243637 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh"] Nov 02 22:31:45.244196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.244181 4309 cpu_manager.go:414] "containerMap: removing container" podUID="5ccdc497-94b0-4fe4-951c-84616411c6e9" containerName="revoke-keys" Nov 02 22:31:45.244256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.244198 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ccdc497-94b0-4fe4-951c-84616411c6e9" containerName="revoke-keys" Nov 02 22:31:45.244331 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.244317 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ccdc497-94b0-4fe4-951c-84616411c6e9" containerName="revoke-keys" Nov 02 22:31:45.247993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.247969 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.252368 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.252333 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:31:45.253089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.252966 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:31:45.253089 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.253052 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:31:45.253427 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.253404 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:31:45.255078 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.255054 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh"] Nov 02 22:31:45.289697 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.289652 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fffq\" (UniqueName: \"kubernetes.io/projected/85469ba4-e983-42c1-b34b-c8f761fba381-kube-api-access-9fffq\") pod \"maas-api-revoke-sub-e2fc9c5b-f74fh\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.289890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.289771 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f74fh\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.289890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.289819 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f74fh\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.391110 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.391068 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fffq\" (UniqueName: \"kubernetes.io/projected/85469ba4-e983-42c1-b34b-c8f761fba381-kube-api-access-9fffq\") pod \"maas-api-revoke-sub-e2fc9c5b-f74fh\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.391296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.391147 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f74fh\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.391296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.391194 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f74fh\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.391946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.391922 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f74fh\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.393431 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.393403 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-f74fh\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.399116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.399093 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fffq\" (UniqueName: \"kubernetes.io/projected/85469ba4-e983-42c1-b34b-c8f761fba381-kube-api-access-9fffq\") pod \"maas-api-revoke-sub-e2fc9c5b-f74fh\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.564661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.564625 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:45.842154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:45.842054 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh"] Nov 02 22:31:45.844809 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:31:45.844779 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85469ba4_e983_42c1_b34b_c8f761fba381.slice/crio-074f8d033531f5a95268983bdd616bd7a44320ef57c96c5e71d19fcfd1d4da00 WatchSource:0}: Error finding container 074f8d033531f5a95268983bdd616bd7a44320ef57c96c5e71d19fcfd1d4da00: Status 404 returned error can't find the container with id 074f8d033531f5a95268983bdd616bd7a44320ef57c96c5e71d19fcfd1d4da00 Nov 02 22:31:46.298558 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:46.298525 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" event={"ID":"85469ba4-e983-42c1-b34b-c8f761fba381","Type":"ContainerStarted","Data":"a64e119ac81510d870b5bb1064278075f0717f78983ea66a823ff52d36a00f1d"} Nov 02 22:31:46.298999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:46.298586 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" event={"ID":"85469ba4-e983-42c1-b34b-c8f761fba381","Type":"ContainerStarted","Data":"074f8d033531f5a95268983bdd616bd7a44320ef57c96c5e71d19fcfd1d4da00"} Nov 02 22:31:46.316024 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:46.315973 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" podStartSLOduration=1.315960125 podStartE2EDuration="1.315960125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:46.312617481 +0000 UTC m=+2034.250730698" watchObservedRunningTime="2026-11-02 22:31:46.315960125 +0000 UTC m=+2034.254073342" Nov 02 22:31:47.306217 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:47.306181 4309 generic.go:356] "Generic (PLEG): container finished" podID="85469ba4-e983-42c1-b34b-c8f761fba381" containerID="a64e119ac81510d870b5bb1064278075f0717f78983ea66a823ff52d36a00f1d" exitCode=0 Nov 02 22:31:47.306649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:47.306274 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" event={"ID":"85469ba4-e983-42c1-b34b-c8f761fba381","Type":"ContainerDied","Data":"a64e119ac81510d870b5bb1064278075f0717f78983ea66a823ff52d36a00f1d"} Nov 02 22:31:48.456353 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.456326 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:48.523065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.523037 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85469ba4-e983-42c1-b34b-c8f761fba381-kube-api-access-9fffq\" (UniqueName: \"kubernetes.io/projected/85469ba4-e983-42c1-b34b-c8f761fba381-kube-api-access-9fffq\") pod \"85469ba4-e983-42c1-b34b-c8f761fba381\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " Nov 02 22:31:48.523234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.523070 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-xks-ca\") pod \"85469ba4-e983-42c1-b34b-c8f761fba381\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " Nov 02 22:31:48.523234 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.523156 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-openshift-ca\") pod \"85469ba4-e983-42c1-b34b-c8f761fba381\" (UID: \"85469ba4-e983-42c1-b34b-c8f761fba381\") " Nov 02 22:31:48.523535 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.523514 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-openshift-ca" pod "85469ba4-e983-42c1-b34b-c8f761fba381" (UID: "85469ba4-e983-42c1-b34b-c8f761fba381"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:48.525272 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.525253 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-xks-ca" pod "85469ba4-e983-42c1-b34b-c8f761fba381" (UID: "85469ba4-e983-42c1-b34b-c8f761fba381"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:48.525357 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.525302 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85469ba4-e983-42c1-b34b-c8f761fba381-kube-api-access-9fffq" pod "85469ba4-e983-42c1-b34b-c8f761fba381" (UID: "85469ba4-e983-42c1-b34b-c8f761fba381"). InnerVolumeSpecName "kube-api-access-9fffq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:48.624002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.623929 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:48.624002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.623954 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fffq\" (UniqueName: \"kubernetes.io/projected/85469ba4-e983-42c1-b34b-c8f761fba381-kube-api-access-9fffq\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:48.624002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:48.623964 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85469ba4-e983-42c1-b34b-c8f761fba381-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:49.320738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:49.320702 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" Nov 02 22:31:49.320738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:49.320726 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh" event={"ID":"85469ba4-e983-42c1-b34b-c8f761fba381","Type":"ContainerDied","Data":"074f8d033531f5a95268983bdd616bd7a44320ef57c96c5e71d19fcfd1d4da00"} Nov 02 22:31:49.321010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:49.320779 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="074f8d033531f5a95268983bdd616bd7a44320ef57c96c5e71d19fcfd1d4da00" Nov 02 22:31:55.439957 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.439922 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x"] Nov 02 22:31:55.440510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.440367 4309 cpu_manager.go:414] "containerMap: removing container" podUID="85469ba4-e983-42c1-b34b-c8f761fba381" containerName="revoke-keys" Nov 02 22:31:55.440510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.440378 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85469ba4-e983-42c1-b34b-c8f761fba381" containerName="revoke-keys" Nov 02 22:31:55.440510 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.440481 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="85469ba4-e983-42c1-b34b-c8f761fba381" containerName="revoke-keys" Nov 02 22:31:55.443830 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.443810 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.448596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.447845 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:31:55.448596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.447968 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:31:55.448596 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.448061 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:31:55.449158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.449136 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:31:55.451280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.451255 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x"] Nov 02 22:31:55.498513 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.498484 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-vk29x\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.498696 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.498524 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-vk29x\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.498754 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.498708 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7b9\" (UniqueName: \"kubernetes.io/projected/1f99e21c-cf67-420c-b738-485cf00ba3db-kube-api-access-ql7b9\") pod \"maas-api-revoke-sub-5a29efb3-vk29x\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.600250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.600220 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-vk29x\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.600250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.600259 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-vk29x\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.600499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.600328 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7b9\" (UniqueName: \"kubernetes.io/projected/1f99e21c-cf67-420c-b738-485cf00ba3db-kube-api-access-ql7b9\") pod \"maas-api-revoke-sub-5a29efb3-vk29x\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.601062 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.601035 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-vk29x\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.602669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.602646 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-vk29x\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.608245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.608219 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7b9\" (UniqueName: \"kubernetes.io/projected/1f99e21c-cf67-420c-b738-485cf00ba3db-kube-api-access-ql7b9\") pod \"maas-api-revoke-sub-5a29efb3-vk29x\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:55.763540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:55.763459 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:56.047836 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:56.047807 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x"] Nov 02 22:31:56.049352 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:31:56.049327 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f99e21c_cf67_420c_b738_485cf00ba3db.slice/crio-1de23428e9d605b261e06328604115ce883bab61a2a0cb6790a9659b21f5ef63 WatchSource:0}: Error finding container 1de23428e9d605b261e06328604115ce883bab61a2a0cb6790a9659b21f5ef63: Status 404 returned error can't find the container with id 1de23428e9d605b261e06328604115ce883bab61a2a0cb6790a9659b21f5ef63 Nov 02 22:31:56.374634 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:56.374596 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" event={"ID":"1f99e21c-cf67-420c-b738-485cf00ba3db","Type":"ContainerStarted","Data":"5c687f3d287eba8d971d6a9d99c0a48f8731abb730d3a61f593efc70867ea910"} Nov 02 22:31:56.374835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:56.374660 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" event={"ID":"1f99e21c-cf67-420c-b738-485cf00ba3db","Type":"ContainerStarted","Data":"1de23428e9d605b261e06328604115ce883bab61a2a0cb6790a9659b21f5ef63"} Nov 02 22:31:56.390294 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:56.390250 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" podStartSLOduration=1.390237248 podStartE2EDuration="1.390237248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:31:56.388886477 +0000 UTC m=+2044.326999695" watchObservedRunningTime="2026-11-02 22:31:56.390237248 +0000 UTC m=+2044.328350456" Nov 02 22:31:57.403557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:57.403045 4309 generic.go:356] "Generic (PLEG): container finished" podID="1f99e21c-cf67-420c-b738-485cf00ba3db" containerID="5c687f3d287eba8d971d6a9d99c0a48f8731abb730d3a61f593efc70867ea910" exitCode=0 Nov 02 22:31:57.403557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:57.403176 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" event={"ID":"1f99e21c-cf67-420c-b738-485cf00ba3db","Type":"ContainerDied","Data":"5c687f3d287eba8d971d6a9d99c0a48f8731abb730d3a61f593efc70867ea910"} Nov 02 22:31:58.044699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:58.040719 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58"] Nov 02 22:31:58.044699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:58.041719 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25396947-b2f58"] Nov 02 22:31:58.227710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:58.226614 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9614ca9-3441-43e7-a9f4-278ebff05cb6" path="/var/lib/kubelet/pods/e9614ca9-3441-43e7-a9f4-278ebff05cb6/volumes" Nov 02 22:31:58.940668 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:58.940643 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:59.015598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.015558 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f99e21c-cf67-420c-b738-485cf00ba3db-kube-api-access-ql7b9\" (UniqueName: \"kubernetes.io/projected/1f99e21c-cf67-420c-b738-485cf00ba3db-kube-api-access-ql7b9\") pod \"1f99e21c-cf67-420c-b738-485cf00ba3db\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " Nov 02 22:31:59.015823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.015625 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-openshift-ca\") pod \"1f99e21c-cf67-420c-b738-485cf00ba3db\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " Nov 02 22:31:59.015823 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.015713 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-xks-ca\") pod \"1f99e21c-cf67-420c-b738-485cf00ba3db\" (UID: \"1f99e21c-cf67-420c-b738-485cf00ba3db\") " Nov 02 22:31:59.017229 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.017198 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-openshift-ca" pod "1f99e21c-cf67-420c-b738-485cf00ba3db" (UID: "1f99e21c-cf67-420c-b738-485cf00ba3db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:31:59.023277 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.023242 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f99e21c-cf67-420c-b738-485cf00ba3db-kube-api-access-ql7b9" pod "1f99e21c-cf67-420c-b738-485cf00ba3db" (UID: "1f99e21c-cf67-420c-b738-485cf00ba3db"). InnerVolumeSpecName "kube-api-access-ql7b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:31:59.025168 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.025140 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-xks-ca" pod "1f99e21c-cf67-420c-b738-485cf00ba3db" (UID: "1f99e21c-cf67-420c-b738-485cf00ba3db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:31:59.117022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.116976 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql7b9\" (UniqueName: \"kubernetes.io/projected/1f99e21c-cf67-420c-b738-485cf00ba3db-kube-api-access-ql7b9\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:59.117022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.117008 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:59.117022 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.117018 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f99e21c-cf67-420c-b738-485cf00ba3db-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:31:59.443125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.443094 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" Nov 02 22:31:59.443281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.443124 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x" event={"ID":"1f99e21c-cf67-420c-b738-485cf00ba3db","Type":"ContainerDied","Data":"1de23428e9d605b261e06328604115ce883bab61a2a0cb6790a9659b21f5ef63"} Nov 02 22:31:59.443281 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:31:59.443165 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1de23428e9d605b261e06328604115ce883bab61a2a0cb6790a9659b21f5ef63" Nov 02 22:32:00.630711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.630662 4309 scope.go:122] "RemoveContainer" containerID="d0214a4f9d8c4605d483f156a398aad49583de483ec6fd89af896b228f1a7c91" Nov 02 22:32:00.640457 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.640430 4309 scope.go:122] "RemoveContainer" containerID="16fd2e028993bb03f171e96fb78dc5a16fddac095010727769dc327d0bc521ff" Nov 02 22:32:00.648721 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.648697 4309 scope.go:122] "RemoveContainer" containerID="510abe3c6574511a6332c972d96cc24fb8963ffe06d657925e02185c88ab287a" Nov 02 22:32:00.656779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.656756 4309 scope.go:122] "RemoveContainer" containerID="d82ee1eb13997e90abcd8b6188300e947c100e9f42423c9253ad0459366cf99f" Nov 02 22:32:00.664909 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.664891 4309 scope.go:122] "RemoveContainer" containerID="feb8cee33d6448211d0713076e53c379c9681622e24d4eefcc7cd65fbdb9e126" Nov 02 22:32:00.674045 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.674025 4309 scope.go:122] "RemoveContainer" containerID="3f0cbe17ab816caf7ad69bb8b7b2d4023d17d7b55c52ab6375bc135f40b0edc4" Nov 02 22:32:00.688486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.688468 4309 scope.go:122] "RemoveContainer" containerID="e4600c1903344d21a2c4fa68c7b22f4784aa6af00c6f58bce6d5edc830e97313" Nov 02 22:32:00.696844 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.696816 4309 scope.go:122] "RemoveContainer" containerID="732750424ae5d507b52c5d919817fd422eeb50a93a4e7f4e9d42d42d209d9c55" Nov 02 22:32:00.704928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.704904 4309 scope.go:122] "RemoveContainer" containerID="f6a34f2d04fd4ef688d1c2a4aead899b66f783568558981ead62c10718b80311" Nov 02 22:32:00.712820 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.712800 4309 scope.go:122] "RemoveContainer" containerID="eefacf8613cb8188071cd7fc04713c4baa956f10909df506717bc0b14cb9257c" Nov 02 22:32:00.720536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.720519 4309 scope.go:122] "RemoveContainer" containerID="e892a5da4173b8b4a3f0a21ca03240779b28399da815c17cfbb0e48a78009d86" Nov 02 22:32:00.728341 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:00.728316 4309 scope.go:122] "RemoveContainer" containerID="d2e704953dc6afd0a4448cee62d60d76e44d5cf72db10970ceddb96a7220e895" Nov 02 22:32:01.799252 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:01.799219 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn"] Nov 02 22:32:01.799797 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:01.799434 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" podUID="60ee5122-b354-4878-ad0b-bed64f19aad9" containerName="discovery" containerID="cri-o://1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f" gracePeriod=30 Nov 02 22:32:02.053650 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.053584 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:32:02.145145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.145102 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-token\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-token\") pod \"60ee5122-b354-4878-ad0b-bed64f19aad9\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " Nov 02 22:32:02.145346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.145176 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-ca-configmap\") pod \"60ee5122-b354-4878-ad0b-bed64f19aad9\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " Nov 02 22:32:02.145346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.145196 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-kubeconfig\") pod \"60ee5122-b354-4878-ad0b-bed64f19aad9\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " Nov 02 22:32:02.145346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.145231 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-kube-api-access-dtc6j\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-kube-api-access-dtc6j\") pod \"60ee5122-b354-4878-ad0b-bed64f19aad9\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " Nov 02 22:32:02.145551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.145376 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-dns-cert\") pod \"60ee5122-b354-4878-ad0b-bed64f19aad9\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " Nov 02 22:32:02.145551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.145445 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-cacerts\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-cacerts\") pod \"60ee5122-b354-4878-ad0b-bed64f19aad9\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " Nov 02 22:32:02.145666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.145556 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ee5122-b354-4878-ad0b-bed64f19aad9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee5122-b354-4878-ad0b-bed64f19aad9-local-certs\") pod \"60ee5122-b354-4878-ad0b-bed64f19aad9\" (UID: \"60ee5122-b354-4878-ad0b-bed64f19aad9\") " Nov 02 22:32:02.145666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.145640 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-ca-configmap" pod "60ee5122-b354-4878-ad0b-bed64f19aad9" (UID: "60ee5122-b354-4878-ad0b-bed64f19aad9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:02.146084 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.146059 4309 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-ca-configmap\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:02.147845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.147802 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-token" pod "60ee5122-b354-4878-ad0b-bed64f19aad9" (UID: "60ee5122-b354-4878-ad0b-bed64f19aad9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:02.147845 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.147820 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-cacerts" pod "60ee5122-b354-4878-ad0b-bed64f19aad9" (UID: "60ee5122-b354-4878-ad0b-bed64f19aad9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:02.148061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.148022 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-kubeconfig" pod "60ee5122-b354-4878-ad0b-bed64f19aad9" (UID: "60ee5122-b354-4878-ad0b-bed64f19aad9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:02.148061 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.148055 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-dns-cert" pod "60ee5122-b354-4878-ad0b-bed64f19aad9" (UID: "60ee5122-b354-4878-ad0b-bed64f19aad9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:02.148218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.148101 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ee5122-b354-4878-ad0b-bed64f19aad9-local-certs" pod "60ee5122-b354-4878-ad0b-bed64f19aad9" (UID: "60ee5122-b354-4878-ad0b-bed64f19aad9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:32:02.148218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.148143 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-kube-api-access-dtc6j" pod "60ee5122-b354-4878-ad0b-bed64f19aad9" (UID: "60ee5122-b354-4878-ad0b-bed64f19aad9"). InnerVolumeSpecName "kube-api-access-dtc6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:02.247634 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.247602 4309 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-token\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:02.247634 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.247629 4309 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-kubeconfig\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:02.247634 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.247641 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtc6j\" (UniqueName: \"kubernetes.io/projected/60ee5122-b354-4878-ad0b-bed64f19aad9-kube-api-access-dtc6j\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:02.247889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.247650 4309 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-istio-csr-dns-cert\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:02.247889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.247659 4309 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/60ee5122-b354-4878-ad0b-bed64f19aad9-cacerts\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:02.247889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.247668 4309 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/60ee5122-b354-4878-ad0b-bed64f19aad9-local-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:02.463572 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.463533 4309 generic.go:356] "Generic (PLEG): container finished" podID="60ee5122-b354-4878-ad0b-bed64f19aad9" containerID="1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f" exitCode=0 Nov 02 22:32:02.463800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.463578 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" Nov 02 22:32:02.463800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.463602 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" event={"ID":"60ee5122-b354-4878-ad0b-bed64f19aad9","Type":"ContainerDied","Data":"1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f"} Nov 02 22:32:02.463800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.463658 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn" event={"ID":"60ee5122-b354-4878-ad0b-bed64f19aad9","Type":"ContainerDied","Data":"260c5bed38323edac2f271f7d8b09de4ef4d28b7c3fc8301beb2a5fbdb918666"} Nov 02 22:32:02.463800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.463712 4309 scope.go:122] "RemoveContainer" containerID="1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f" Nov 02 22:32:02.474602 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.474581 4309 scope.go:122] "RemoveContainer" containerID="1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f" Nov 02 22:32:02.474937 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:32:02.474918 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f\": container with ID starting with 1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f not found: ID does not exist" containerID="1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f" Nov 02 22:32:02.475033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.474947 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f"} err="failed to get container status \"1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f\": rpc error: code = NotFound desc = could not find container \"1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f\": container with ID starting with 1e8a33e2430b4247fe57f9282dcd8b4b88e03de8ea86cff13e5fc7e13c5c9c1f not found: ID does not exist" Nov 02 22:32:02.481989 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.481967 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn"] Nov 02 22:32:02.485278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:02.485256 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-n6rpn"] Nov 02 22:32:04.187887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:04.187857 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ee5122-b354-4878-ad0b-bed64f19aad9" path="/var/lib/kubelet/pods/60ee5122-b354-4878-ad0b-bed64f19aad9/volumes" Nov 02 22:32:10.864536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.864500 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j"] Nov 02 22:32:10.868900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.868874 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fnl7j"] Nov 02 22:32:10.885291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.885262 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll"] Nov 02 22:32:10.885744 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.885731 4309 cpu_manager.go:414] "containerMap: removing container" podUID="1f99e21c-cf67-420c-b738-485cf00ba3db" containerName="revoke-keys" Nov 02 22:32:10.885808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.885746 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f99e21c-cf67-420c-b738-485cf00ba3db" containerName="revoke-keys" Nov 02 22:32:10.885808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.885775 4309 cpu_manager.go:414] "containerMap: removing container" podUID="60ee5122-b354-4878-ad0b-bed64f19aad9" containerName="discovery" Nov 02 22:32:10.885808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.885780 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ee5122-b354-4878-ad0b-bed64f19aad9" containerName="discovery" Nov 02 22:32:10.885907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.885850 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ee5122-b354-4878-ad0b-bed64f19aad9" containerName="discovery" Nov 02 22:32:10.885907 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.885863 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f99e21c-cf67-420c-b738-485cf00ba3db" containerName="revoke-keys" Nov 02 22:32:10.890478 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.890463 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:10.894978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.894818 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:32:10.894978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.894927 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:32:10.896120 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.896068 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:32:10.896335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.896117 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:32:10.896335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.896268 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll"] Nov 02 22:32:10.932621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.932587 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvmll\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:10.932795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.932638 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvmll\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:10.932795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:10.932662 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlpvp\" (UniqueName: \"kubernetes.io/projected/58d05019-dc67-4db4-9432-65ed0d584a6c-kube-api-access-hlpvp\") pod \"maas-api-revoke-sub-3d4156f1-jvmll\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:11.033249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:11.033214 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvmll\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:11.033430 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:11.033265 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvmll\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:11.033430 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:11.033290 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlpvp\" (UniqueName: \"kubernetes.io/projected/58d05019-dc67-4db4-9432-65ed0d584a6c-kube-api-access-hlpvp\") pod \"maas-api-revoke-sub-3d4156f1-jvmll\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:11.033867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:11.033844 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvmll\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:11.035741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:11.035722 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jvmll\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:11.040949 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:11.040920 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlpvp\" (UniqueName: \"kubernetes.io/projected/58d05019-dc67-4db4-9432-65ed0d584a6c-kube-api-access-hlpvp\") pod \"maas-api-revoke-sub-3d4156f1-jvmll\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:11.207094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:11.207008 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:11.480623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:11.480595 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll"] Nov 02 22:32:11.482302 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:32:11.482273 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d05019_dc67_4db4_9432_65ed0d584a6c.slice/crio-891eebabd6fd648d71e27e9d7e2a9c1cb7efdd24978a1b9202e97fa172e5c367 WatchSource:0}: Error finding container 891eebabd6fd648d71e27e9d7e2a9c1cb7efdd24978a1b9202e97fa172e5c367: Status 404 returned error can't find the container with id 891eebabd6fd648d71e27e9d7e2a9c1cb7efdd24978a1b9202e97fa172e5c367 Nov 02 22:32:11.525146 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:11.525115 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" event={"ID":"58d05019-dc67-4db4-9432-65ed0d584a6c","Type":"ContainerStarted","Data":"891eebabd6fd648d71e27e9d7e2a9c1cb7efdd24978a1b9202e97fa172e5c367"} Nov 02 22:32:11.599748 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:32:11.599713 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:32:12.188903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:12.188869 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ccdc497-94b0-4fe4-951c-84616411c6e9" path="/var/lib/kubelet/pods/5ccdc497-94b0-4fe4-951c-84616411c6e9/volumes" Nov 02 22:32:12.531701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:12.531598 4309 generic.go:356] "Generic (PLEG): container finished" podID="58d05019-dc67-4db4-9432-65ed0d584a6c" containerID="affa8fa77e053d8dff760da68e18630345b482e323ded74fa2a5ea794901a654" exitCode=0 Nov 02 22:32:12.531848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:12.531694 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" event={"ID":"58d05019-dc67-4db4-9432-65ed0d584a6c","Type":"ContainerDied","Data":"affa8fa77e053d8dff760da68e18630345b482e323ded74fa2a5ea794901a654"} Nov 02 22:32:13.680184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.680161 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:13.756047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.756015 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-xks-ca\") pod \"58d05019-dc67-4db4-9432-65ed0d584a6c\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " Nov 02 22:32:13.756256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.756150 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d05019-dc67-4db4-9432-65ed0d584a6c-kube-api-access-hlpvp\" (UniqueName: \"kubernetes.io/projected/58d05019-dc67-4db4-9432-65ed0d584a6c-kube-api-access-hlpvp\") pod \"58d05019-dc67-4db4-9432-65ed0d584a6c\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " Nov 02 22:32:13.756256 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.756222 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-openshift-ca\") pod \"58d05019-dc67-4db4-9432-65ed0d584a6c\" (UID: \"58d05019-dc67-4db4-9432-65ed0d584a6c\") " Nov 02 22:32:13.756545 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.756523 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-openshift-ca" pod "58d05019-dc67-4db4-9432-65ed0d584a6c" (UID: "58d05019-dc67-4db4-9432-65ed0d584a6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:32:13.758125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.758105 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d05019-dc67-4db4-9432-65ed0d584a6c-kube-api-access-hlpvp" pod "58d05019-dc67-4db4-9432-65ed0d584a6c" (UID: "58d05019-dc67-4db4-9432-65ed0d584a6c"). InnerVolumeSpecName "kube-api-access-hlpvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:32:13.758215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.758195 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-xks-ca" pod "58d05019-dc67-4db4-9432-65ed0d584a6c" (UID: "58d05019-dc67-4db4-9432-65ed0d584a6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:32:13.857291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.857259 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlpvp\" (UniqueName: \"kubernetes.io/projected/58d05019-dc67-4db4-9432-65ed0d584a6c-kube-api-access-hlpvp\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.857291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.857289 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:13.857291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:13.857298 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58d05019-dc67-4db4-9432-65ed0d584a6c-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:32:14.544486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:14.544405 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" Nov 02 22:32:14.544486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:14.544452 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll" event={"ID":"58d05019-dc67-4db4-9432-65ed0d584a6c","Type":"ContainerDied","Data":"891eebabd6fd648d71e27e9d7e2a9c1cb7efdd24978a1b9202e97fa172e5c367"} Nov 02 22:32:14.544693 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:14.544506 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="891eebabd6fd648d71e27e9d7e2a9c1cb7efdd24978a1b9202e97fa172e5c367" Nov 02 22:32:41.595250 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:32:41.595202 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:32:48.022638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:48.022604 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s"] Nov 02 22:32:48.025105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:48.025076 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rfz4s"] Nov 02 22:32:48.187257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:48.187215 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0522ba-9931-4623-982a-c6925028c7c2" path="/var/lib/kubelet/pods/4a0522ba-9931-4623-982a-c6925028c7c2/volumes" Nov 02 22:32:56.558962 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:56.558927 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:32:56.568353 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:32:56.568319 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:33:00.803670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.803640 4309 scope.go:122] "RemoveContainer" containerID="dddba56ce3287850d05de51acf23349e5b5809dfe7d6b306421b1abb01800646" Nov 02 22:33:00.813155 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.813134 4309 scope.go:122] "RemoveContainer" containerID="c401da5c9200c63e7ad6f48ca2fef2047bcd84905701ed3c242b4add650a6a66" Nov 02 22:33:00.821463 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.821447 4309 scope.go:122] "RemoveContainer" containerID="7325174642f4b3b473592e4389d4b33f423148c8151ab1e62bdcf3243309a92f" Nov 02 22:33:00.829409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.829392 4309 scope.go:122] "RemoveContainer" containerID="96cca41669ac0d6d4dedb66a8ff308a5da45c5a925ef1d041478657168e1a7a4" Nov 02 22:33:00.837149 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.837132 4309 scope.go:122] "RemoveContainer" containerID="04baf2c2e945aaeb7b96f582f3dbbec981f6fc2d591819c6451e211b74b5ddc0" Nov 02 22:33:00.845546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.845521 4309 scope.go:122] "RemoveContainer" containerID="9645bb9ffd851022f7ec215fda7d688a87c2a297e0ae8dbf835c6f3cf413bc4a" Nov 02 22:33:00.853189 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.853170 4309 scope.go:122] "RemoveContainer" containerID="e3d4069f8ecf89fbbee220fb783c702c5ceb0bd929995ad0b3e1547f76ea1415" Nov 02 22:33:00.861113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.861096 4309 scope.go:122] "RemoveContainer" containerID="fd24133c356812ec04684cdab238b8b7d34fbb8abe367ac98f60522f18dc529a" Nov 02 22:33:00.869785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.869764 4309 scope.go:122] "RemoveContainer" containerID="ec1f13486e06b755d50742d8a158d642f7afc426500c2a1a9cc8cb00f39a4675" Nov 02 22:33:00.878129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.878106 4309 scope.go:122] "RemoveContainer" containerID="b7c28d5c2e9b736c1bb32e8e770fb62cbe27eb3d4105afbf3ae6ff56bfb01d98" Nov 02 22:33:00.887291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:00.887273 4309 scope.go:122] "RemoveContainer" containerID="30e5c7728d03fd872b5756e93a5111abd5c399b23ae5f557355a3b542d83f68e" Nov 02 22:33:11.594913 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:33:11.594829 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:33:19.710716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.710670 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj"] Nov 02 22:33:19.711220 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.711112 4309 cpu_manager.go:414] "containerMap: removing container" podUID="58d05019-dc67-4db4-9432-65ed0d584a6c" containerName="revoke-keys" Nov 02 22:33:19.711220 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.711123 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d05019-dc67-4db4-9432-65ed0d584a6c" containerName="revoke-keys" Nov 02 22:33:19.711220 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.711210 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d05019-dc67-4db4-9432-65ed0d584a6c" containerName="revoke-keys" Nov 02 22:33:19.714300 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.714285 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:19.722171 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.722138 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:33:19.722328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.722244 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:33:19.723148 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.723123 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj"] Nov 02 22:33:19.780415 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.780378 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:33:19.780736 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.780699 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:33:19.865741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.865696 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51179b85-hbftj\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:19.865921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.865772 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51179b85-hbftj\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:19.865921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.865887 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds2rw\" (UniqueName: \"kubernetes.io/projected/dcf653ce-c37f-4098-8d11-79c9f16d0545-kube-api-access-ds2rw\") pod \"maas-api-revoke-sub-51179b85-hbftj\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:19.967468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.967376 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds2rw\" (UniqueName: \"kubernetes.io/projected/dcf653ce-c37f-4098-8d11-79c9f16d0545-kube-api-access-ds2rw\") pod \"maas-api-revoke-sub-51179b85-hbftj\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:19.967468 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.967462 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51179b85-hbftj\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:19.967751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.967481 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51179b85-hbftj\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:19.968147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.968112 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51179b85-hbftj\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:19.969978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.969956 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51179b85-hbftj\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:19.975740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:19.975707 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds2rw\" (UniqueName: \"kubernetes.io/projected/dcf653ce-c37f-4098-8d11-79c9f16d0545-kube-api-access-ds2rw\") pod \"maas-api-revoke-sub-51179b85-hbftj\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:20.028864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:20.028831 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:20.302388 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:20.302355 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj"] Nov 02 22:33:20.303651 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:33:20.303628 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf653ce_c37f_4098_8d11_79c9f16d0545.slice/crio-b6f910e5ea9e479d277052760894b1261f5a96fd066c6f363937148c37e6bd6c WatchSource:0}: Error finding container b6f910e5ea9e479d277052760894b1261f5a96fd066c6f363937148c37e6bd6c: Status 404 returned error can't find the container with id b6f910e5ea9e479d277052760894b1261f5a96fd066c6f363937148c37e6bd6c Nov 02 22:33:21.010239 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:21.010146 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" event={"ID":"dcf653ce-c37f-4098-8d11-79c9f16d0545","Type":"ContainerStarted","Data":"d8510f1c45bc44c73f987620582f321a0e35f29666e8780e1ce9901ef447b7b5"} Nov 02 22:33:21.010742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:21.010240 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" event={"ID":"dcf653ce-c37f-4098-8d11-79c9f16d0545","Type":"ContainerStarted","Data":"b6f910e5ea9e479d277052760894b1261f5a96fd066c6f363937148c37e6bd6c"} Nov 02 22:33:21.026666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:21.026598 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" podStartSLOduration=2.026580824 podStartE2EDuration="2.026580824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:33:21.0254096 +0000 UTC m=+2128.963522820" watchObservedRunningTime="2026-11-02 22:33:21.026580824 +0000 UTC m=+2128.964694046" Nov 02 22:33:22.018432 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:22.018397 4309 generic.go:356] "Generic (PLEG): container finished" podID="dcf653ce-c37f-4098-8d11-79c9f16d0545" containerID="d8510f1c45bc44c73f987620582f321a0e35f29666e8780e1ce9901ef447b7b5" exitCode=0 Nov 02 22:33:22.018938 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:22.018476 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" event={"ID":"dcf653ce-c37f-4098-8d11-79c9f16d0545","Type":"ContainerDied","Data":"d8510f1c45bc44c73f987620582f321a0e35f29666e8780e1ce9901ef447b7b5"} Nov 02 22:33:23.162116 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.162088 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:23.301869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.301790 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-openshift-ca\") pod \"dcf653ce-c37f-4098-8d11-79c9f16d0545\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " Nov 02 22:33:23.301869 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.301866 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-xks-ca\") pod \"dcf653ce-c37f-4098-8d11-79c9f16d0545\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " Nov 02 22:33:23.302063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.301894 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf653ce-c37f-4098-8d11-79c9f16d0545-kube-api-access-ds2rw\" (UniqueName: \"kubernetes.io/projected/dcf653ce-c37f-4098-8d11-79c9f16d0545-kube-api-access-ds2rw\") pod \"dcf653ce-c37f-4098-8d11-79c9f16d0545\" (UID: \"dcf653ce-c37f-4098-8d11-79c9f16d0545\") " Nov 02 22:33:23.302219 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.302199 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-openshift-ca" pod "dcf653ce-c37f-4098-8d11-79c9f16d0545" (UID: "dcf653ce-c37f-4098-8d11-79c9f16d0545"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:23.302322 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.302299 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:23.304046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.304023 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf653ce-c37f-4098-8d11-79c9f16d0545-kube-api-access-ds2rw" pod "dcf653ce-c37f-4098-8d11-79c9f16d0545" (UID: "dcf653ce-c37f-4098-8d11-79c9f16d0545"). InnerVolumeSpecName "kube-api-access-ds2rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:23.304046 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.304025 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-xks-ca" pod "dcf653ce-c37f-4098-8d11-79c9f16d0545" (UID: "dcf653ce-c37f-4098-8d11-79c9f16d0545"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:23.403519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.403489 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf653ce-c37f-4098-8d11-79c9f16d0545-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:23.403519 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:23.403517 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds2rw\" (UniqueName: \"kubernetes.io/projected/dcf653ce-c37f-4098-8d11-79c9f16d0545-kube-api-access-ds2rw\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:24.030393 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:24.030357 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" Nov 02 22:33:24.030393 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:24.030367 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj" event={"ID":"dcf653ce-c37f-4098-8d11-79c9f16d0545","Type":"ContainerDied","Data":"b6f910e5ea9e479d277052760894b1261f5a96fd066c6f363937148c37e6bd6c"} Nov 02 22:33:24.030629 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:24.030422 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6f910e5ea9e479d277052760894b1261f5a96fd066c6f363937148c37e6bd6c" Nov 02 22:33:30.786981 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.786943 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7"] Nov 02 22:33:30.787450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.787432 4309 cpu_manager.go:414] "containerMap: removing container" podUID="dcf653ce-c37f-4098-8d11-79c9f16d0545" containerName="revoke-keys" Nov 02 22:33:30.787514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.787451 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf653ce-c37f-4098-8d11-79c9f16d0545" containerName="revoke-keys" Nov 02 22:33:30.787560 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.787549 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf653ce-c37f-4098-8d11-79c9f16d0545" containerName="revoke-keys" Nov 02 22:33:30.790975 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.790958 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:30.797734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.797711 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:33:30.798743 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.798714 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7"] Nov 02 22:33:30.799005 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.798985 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:33:30.831481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.831453 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:33:30.831647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.831470 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:33:30.875365 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.875334 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:30.875516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.875447 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:30.875516 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.875478 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c577\" (UniqueName: \"kubernetes.io/projected/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-kube-api-access-5c577\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:30.976749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.976713 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:30.976934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.976783 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:30.976934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.976816 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c577\" (UniqueName: \"kubernetes.io/projected/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-kube-api-access-5c577\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:30.977533 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.977505 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:30.979104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.979081 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:30.984890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:30.984868 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c577\" (UniqueName: \"kubernetes.io/projected/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-kube-api-access-5c577\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:31.106742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:31.106712 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:31.382347 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:31.382309 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7"] Nov 02 22:33:32.075739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:32.075700 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" event={"ID":"71fc4c14-4727-4ac8-a4aa-e45b42cee03a","Type":"ContainerStarted","Data":"97d6633a9ae6922b3a332a6f6ea85528c94c91432f8691a265ad2711412d7c36"} Nov 02 22:33:32.076183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:32.075760 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" event={"ID":"71fc4c14-4727-4ac8-a4aa-e45b42cee03a","Type":"ContainerStarted","Data":"2cffdb7325fe224e86e63bd11ab34df0690bb7cf5fbe5641da68cb605f59239e"} Nov 02 22:33:32.091068 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:32.090972 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" podStartSLOduration=2.090954075 podStartE2EDuration="2.090954075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:33:32.090784512 +0000 UTC m=+2140.028897730" watchObservedRunningTime="2026-11-02 22:33:32.090954075 +0000 UTC m=+2140.029067294" Nov 02 22:33:33.082868 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:33.082834 4309 generic.go:356] "Generic (PLEG): container finished" podID="71fc4c14-4727-4ac8-a4aa-e45b42cee03a" containerID="97d6633a9ae6922b3a332a6f6ea85528c94c91432f8691a265ad2711412d7c36" exitCode=0 Nov 02 22:33:33.083258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:33.082929 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" event={"ID":"71fc4c14-4727-4ac8-a4aa-e45b42cee03a","Type":"ContainerDied","Data":"97d6633a9ae6922b3a332a6f6ea85528c94c91432f8691a265ad2711412d7c36"} Nov 02 22:33:34.224063 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.224036 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:34.312262 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.312227 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-openshift-ca\") pod \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " Nov 02 22:33:34.312448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.312289 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-kube-api-access-5c577\" (UniqueName: \"kubernetes.io/projected/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-kube-api-access-5c577\") pod \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " Nov 02 22:33:34.312448 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.312414 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-xks-ca\") pod \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\" (UID: \"71fc4c14-4727-4ac8-a4aa-e45b42cee03a\") " Nov 02 22:33:34.312705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.312656 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-openshift-ca" pod "71fc4c14-4727-4ac8-a4aa-e45b42cee03a" (UID: "71fc4c14-4727-4ac8-a4aa-e45b42cee03a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:34.312896 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.312872 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.314499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.314475 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-kube-api-access-5c577" pod "71fc4c14-4727-4ac8-a4aa-e45b42cee03a" (UID: "71fc4c14-4727-4ac8-a4aa-e45b42cee03a"). InnerVolumeSpecName "kube-api-access-5c577". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:34.314499 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.314488 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-xks-ca" pod "71fc4c14-4727-4ac8-a4aa-e45b42cee03a" (UID: "71fc4c14-4727-4ac8-a4aa-e45b42cee03a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:34.413584 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.413498 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c577\" (UniqueName: \"kubernetes.io/projected/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-kube-api-access-5c577\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:34.413584 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:34.413526 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71fc4c14-4727-4ac8-a4aa-e45b42cee03a-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:35.094746 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:35.094712 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" Nov 02 22:33:35.094996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:35.094736 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7" event={"ID":"71fc4c14-4727-4ac8-a4aa-e45b42cee03a","Type":"ContainerDied","Data":"2cffdb7325fe224e86e63bd11ab34df0690bb7cf5fbe5641da68cb605f59239e"} Nov 02 22:33:35.094996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:35.094806 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cffdb7325fe224e86e63bd11ab34df0690bb7cf5fbe5641da68cb605f59239e" Nov 02 22:33:39.028000 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:39.027965 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb"] Nov 02 22:33:39.030195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:39.030170 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-9d8zb"] Nov 02 22:33:40.186667 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:40.186633 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2464b6f-1517-42dd-8578-fecc2e76c955" path="/var/lib/kubelet/pods/d2464b6f-1517-42dd-8578-fecc2e76c955/volumes" Nov 02 22:33:41.595826 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:33:41.595785 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:33:41.961060 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:41.960968 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7"] Nov 02 22:33:41.964775 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:41.964750 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b5b20e-4flx7"] Nov 02 22:33:42.187821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:42.187786 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71fc4c14-4727-4ac8-a4aa-e45b42cee03a" path="/var/lib/kubelet/pods/71fc4c14-4727-4ac8-a4aa-e45b42cee03a/volumes" Nov 02 22:33:49.586649 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.586609 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll"] Nov 02 22:33:49.591218 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.591185 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jvmll"] Nov 02 22:33:49.647506 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.647467 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl"] Nov 02 22:33:49.648031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.648011 4309 cpu_manager.go:414] "containerMap: removing container" podUID="71fc4c14-4727-4ac8-a4aa-e45b42cee03a" containerName="revoke-keys" Nov 02 22:33:49.648031 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.648031 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fc4c14-4727-4ac8-a4aa-e45b42cee03a" containerName="revoke-keys" Nov 02 22:33:49.648195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.648186 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fc4c14-4727-4ac8-a4aa-e45b42cee03a" containerName="revoke-keys" Nov 02 22:33:49.651297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.651277 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.655515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.655354 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:33:49.655515 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.655397 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:33:49.656481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.656463 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:33:49.656598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.656491 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:33:49.658747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.658724 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl"] Nov 02 22:33:49.791339 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.791304 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/64261281-ef0a-4c2e-a9d5-7c9294e61003-kube-api-access-kqtsv\") pod \"maas-api-revoke-sub-3d4156f1-4ljxl\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.791552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.791357 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ljxl\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.791552 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.791480 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ljxl\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.892065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.891979 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ljxl\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.892250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.892072 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/64261281-ef0a-4c2e-a9d5-7c9294e61003-kube-api-access-kqtsv\") pod \"maas-api-revoke-sub-3d4156f1-4ljxl\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.892250 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.892108 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ljxl\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.892742 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.892709 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ljxl\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.894472 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.894447 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4ljxl\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.899416 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.899389 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/64261281-ef0a-4c2e-a9d5-7c9294e61003-kube-api-access-kqtsv\") pod \"maas-api-revoke-sub-3d4156f1-4ljxl\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:49.966034 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:49.966002 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:50.187145 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:50.187068 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d05019-dc67-4db4-9432-65ed0d584a6c" path="/var/lib/kubelet/pods/58d05019-dc67-4db4-9432-65ed0d584a6c/volumes" Nov 02 22:33:50.241565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:50.241534 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl"] Nov 02 22:33:50.243377 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:33:50.243348 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64261281_ef0a_4c2e_a9d5_7c9294e61003.slice/crio-93188e640fad40b7ddccc6f54d7cd2e4c392b1bad2b85e4383dbd358ad7105de WatchSource:0}: Error finding container 93188e640fad40b7ddccc6f54d7cd2e4c392b1bad2b85e4383dbd358ad7105de: Status 404 returned error can't find the container with id 93188e640fad40b7ddccc6f54d7cd2e4c392b1bad2b85e4383dbd358ad7105de Nov 02 22:33:51.186661 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:51.186623 4309 generic.go:356] "Generic (PLEG): container finished" podID="64261281-ef0a-4c2e-a9d5-7c9294e61003" containerID="7daedc140c31b2f436f40710ab52d8cdcaee8daa48f92bc4c39387f5ca7ea520" exitCode=0 Nov 02 22:33:51.187106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:51.186710 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" event={"ID":"64261281-ef0a-4c2e-a9d5-7c9294e61003","Type":"ContainerDied","Data":"7daedc140c31b2f436f40710ab52d8cdcaee8daa48f92bc4c39387f5ca7ea520"} Nov 02 22:33:51.187106 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:51.186767 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" event={"ID":"64261281-ef0a-4c2e-a9d5-7c9294e61003","Type":"ContainerStarted","Data":"93188e640fad40b7ddccc6f54d7cd2e4c392b1bad2b85e4383dbd358ad7105de"} Nov 02 22:33:52.331626 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.331598 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:52.416540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.416509 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-xks-ca\") pod \"64261281-ef0a-4c2e-a9d5-7c9294e61003\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " Nov 02 22:33:52.416720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.416629 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-openshift-ca\") pod \"64261281-ef0a-4c2e-a9d5-7c9294e61003\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " Nov 02 22:33:52.416720 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.416716 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64261281-ef0a-4c2e-a9d5-7c9294e61003-kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/64261281-ef0a-4c2e-a9d5-7c9294e61003-kube-api-access-kqtsv\") pod \"64261281-ef0a-4c2e-a9d5-7c9294e61003\" (UID: \"64261281-ef0a-4c2e-a9d5-7c9294e61003\") " Nov 02 22:33:52.416977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.416952 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-openshift-ca" pod "64261281-ef0a-4c2e-a9d5-7c9294e61003" (UID: "64261281-ef0a-4c2e-a9d5-7c9294e61003"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:33:52.418614 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.418585 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-xks-ca" pod "64261281-ef0a-4c2e-a9d5-7c9294e61003" (UID: "64261281-ef0a-4c2e-a9d5-7c9294e61003"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:33:52.418734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.418660 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64261281-ef0a-4c2e-a9d5-7c9294e61003-kube-api-access-kqtsv" pod "64261281-ef0a-4c2e-a9d5-7c9294e61003" (UID: "64261281-ef0a-4c2e-a9d5-7c9294e61003"). InnerVolumeSpecName "kube-api-access-kqtsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:33:52.518447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.518364 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqtsv\" (UniqueName: \"kubernetes.io/projected/64261281-ef0a-4c2e-a9d5-7c9294e61003-kube-api-access-kqtsv\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:52.518447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.518395 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:52.518447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:52.518406 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64261281-ef0a-4c2e-a9d5-7c9294e61003-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:33:53.199007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:53.198978 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" Nov 02 22:33:53.199188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:53.198981 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl" event={"ID":"64261281-ef0a-4c2e-a9d5-7c9294e61003","Type":"ContainerDied","Data":"93188e640fad40b7ddccc6f54d7cd2e4c392b1bad2b85e4383dbd358ad7105de"} Nov 02 22:33:53.199188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:53.199101 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93188e640fad40b7ddccc6f54d7cd2e4c392b1bad2b85e4383dbd358ad7105de" Nov 02 22:33:59.101393 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:59.101356 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck"] Nov 02 22:33:59.110246 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:33:59.110213 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g4sck"] Nov 02 22:34:00.188235 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:00.188206 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf30a29-154a-406d-b4d9-28cac1305e80" path="/var/lib/kubelet/pods/9bf30a29-154a-406d-b4d9-28cac1305e80/volumes" Nov 02 22:34:00.954125 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:00.954090 4309 scope.go:122] "RemoveContainer" containerID="20e1a89e297db627b58c55345239e4c4ed65ad6c44aa28bd1cf70141831bb298" Nov 02 22:34:08.344137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:08.344104 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm"] Nov 02 22:34:08.344580 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:08.344339 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" podUID="1b529273-448c-4adb-9909-5f0e5faa5ca0" containerName="manager" containerID="cri-o://77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042" gracePeriod=10 Nov 02 22:34:08.589261 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:08.589236 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:34:08.700544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:08.700466 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b529273-448c-4adb-9909-5f0e5faa5ca0-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b529273-448c-4adb-9909-5f0e5faa5ca0-extensions-socket-volume\") pod \"1b529273-448c-4adb-9909-5f0e5faa5ca0\" (UID: \"1b529273-448c-4adb-9909-5f0e5faa5ca0\") " Nov 02 22:34:08.700544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:08.700503 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b529273-448c-4adb-9909-5f0e5faa5ca0-kube-api-access-ls8n2\" (UniqueName: \"kubernetes.io/projected/1b529273-448c-4adb-9909-5f0e5faa5ca0-kube-api-access-ls8n2\") pod \"1b529273-448c-4adb-9909-5f0e5faa5ca0\" (UID: \"1b529273-448c-4adb-9909-5f0e5faa5ca0\") " Nov 02 22:34:08.700834 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:08.700812 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b529273-448c-4adb-9909-5f0e5faa5ca0-extensions-socket-volume" pod "1b529273-448c-4adb-9909-5f0e5faa5ca0" (UID: "1b529273-448c-4adb-9909-5f0e5faa5ca0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 22:34:08.702541 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:08.702518 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b529273-448c-4adb-9909-5f0e5faa5ca0-kube-api-access-ls8n2" pod "1b529273-448c-4adb-9909-5f0e5faa5ca0" (UID: "1b529273-448c-4adb-9909-5f0e5faa5ca0"). InnerVolumeSpecName "kube-api-access-ls8n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:34:08.801318 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:08.801286 4309 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b529273-448c-4adb-9909-5f0e5faa5ca0-extensions-socket-volume\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:08.801318 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:08.801317 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls8n2\" (UniqueName: \"kubernetes.io/projected/1b529273-448c-4adb-9909-5f0e5faa5ca0-kube-api-access-ls8n2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:34:09.306898 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:09.306864 4309 generic.go:356] "Generic (PLEG): container finished" podID="1b529273-448c-4adb-9909-5f0e5faa5ca0" containerID="77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042" exitCode=0 Nov 02 22:34:09.307095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:09.306908 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" Nov 02 22:34:09.307095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:09.306912 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" event={"ID":"1b529273-448c-4adb-9909-5f0e5faa5ca0","Type":"ContainerDied","Data":"77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042"} Nov 02 22:34:09.307095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:09.307053 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm" event={"ID":"1b529273-448c-4adb-9909-5f0e5faa5ca0","Type":"ContainerDied","Data":"2f4066d35808a9aba4f0e65b91992d3ac8887c87dbc9d07dec304be1b964015b"} Nov 02 22:34:09.307095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:09.307088 4309 scope.go:122] "RemoveContainer" containerID="77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042" Nov 02 22:34:09.319143 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:09.319124 4309 scope.go:122] "RemoveContainer" containerID="77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042" Nov 02 22:34:09.319417 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:34:09.319396 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042\": container with ID starting with 77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042 not found: ID does not exist" containerID="77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042" Nov 02 22:34:09.319485 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:09.319425 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042"} err="failed to get container status \"77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042\": rpc error: code = NotFound desc = could not find container \"77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042\": container with ID starting with 77b1c50214fca16c89b117831ac411f72652640297c10223eaecae5fc701e042 not found: ID does not exist" Nov 02 22:34:09.330340 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:09.330316 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm"] Nov 02 22:34:09.333517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:09.333498 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4svmm"] Nov 02 22:34:10.020559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:10.020526 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr"] Nov 02 22:34:10.022770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:10.022747 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-5clmr"] Nov 02 22:34:10.186815 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:10.186784 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b529273-448c-4adb-9909-5f0e5faa5ca0" path="/var/lib/kubelet/pods/1b529273-448c-4adb-9909-5f0e5faa5ca0/volumes" Nov 02 22:34:10.187186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:10.187173 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="203decfc-a23e-4b98-a63a-992e8f16e2d7" path="/var/lib/kubelet/pods/203decfc-a23e-4b98-a63a-992e8f16e2d7/volumes" Nov 02 22:34:11.593851 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:34:11.593814 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:34:33.020315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:33.020276 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z"] Nov 02 22:34:33.022409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:33.022387 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-tgk9z"] Nov 02 22:34:34.192838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:34.192792 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96fe7b38-8085-493d-82cb-650451925bb1" path="/var/lib/kubelet/pods/96fe7b38-8085-493d-82cb-650451925bb1/volumes" Nov 02 22:34:41.594811 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:34:41.594728 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:34:43.020643 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:43.020574 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt"] Nov 02 22:34:43.022666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:43.022641 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-tl4pt"] Nov 02 22:34:44.187539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:34:44.187505 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba14f814-d8ca-47ab-9c0f-d5dcf0417294" path="/var/lib/kubelet/pods/ba14f814-d8ca-47ab-9c0f-d5dcf0417294/volumes" Nov 02 22:35:00.982337 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:00.982303 4309 scope.go:122] "RemoveContainer" containerID="2809fb6bb89cb7a3e2a5d0890f5e9c3d38df03a0ff160e9dbec441d1b0510610" Nov 02 22:35:00.991223 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:00.991201 4309 scope.go:122] "RemoveContainer" containerID="7120c409a45b175214e3be24b2fb6db5b95d0d6f1efb7c5129a86f210025f241" Nov 02 22:35:03.025203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:03.025165 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4"] Nov 02 22:35:03.027093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:03.027068 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-vfnc4"] Nov 02 22:35:04.187638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:04.187597 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288fb5ad-2361-44de-ba36-d4ff7a4b0799" path="/var/lib/kubelet/pods/288fb5ad-2361-44de-ba36-d4ff7a4b0799/volumes" Nov 02 22:35:09.426605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.426569 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2"] Nov 02 22:35:09.427127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.427014 4309 cpu_manager.go:414] "containerMap: removing container" podUID="1b529273-448c-4adb-9909-5f0e5faa5ca0" containerName="manager" Nov 02 22:35:09.427127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.427027 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b529273-448c-4adb-9909-5f0e5faa5ca0" containerName="manager" Nov 02 22:35:09.427127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.427041 4309 cpu_manager.go:414] "containerMap: removing container" podUID="64261281-ef0a-4c2e-a9d5-7c9294e61003" containerName="revoke-keys" Nov 02 22:35:09.427127 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.427046 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64261281-ef0a-4c2e-a9d5-7c9294e61003" containerName="revoke-keys" Nov 02 22:35:09.427266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.427134 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b529273-448c-4adb-9909-5f0e5faa5ca0" containerName="manager" Nov 02 22:35:09.427266 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.427142 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="64261281-ef0a-4c2e-a9d5-7c9294e61003" containerName="revoke-keys" Nov 02 22:35:09.431860 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.431842 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:09.443651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.443623 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7kl6p\"" Nov 02 22:35:09.444867 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.444843 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2"] Nov 02 22:35:09.504826 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.504783 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6dn6\" (UniqueName: \"kubernetes.io/projected/aa0e89fc-92fe-4b69-9f63-6e94c05a01e0-kube-api-access-r6dn6\") pod \"kuadrant-operator-controller-manager-678d87766f-hwzg2\" (UID: \"aa0e89fc-92fe-4b69-9f63-6e94c05a01e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:09.505008 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.504949 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa0e89fc-92fe-4b69-9f63-6e94c05a01e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hwzg2\" (UID: \"aa0e89fc-92fe-4b69-9f63-6e94c05a01e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:09.606328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.606291 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6dn6\" (UniqueName: \"kubernetes.io/projected/aa0e89fc-92fe-4b69-9f63-6e94c05a01e0-kube-api-access-r6dn6\") pod \"kuadrant-operator-controller-manager-678d87766f-hwzg2\" (UID: \"aa0e89fc-92fe-4b69-9f63-6e94c05a01e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:09.606538 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.606393 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa0e89fc-92fe-4b69-9f63-6e94c05a01e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hwzg2\" (UID: \"aa0e89fc-92fe-4b69-9f63-6e94c05a01e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:09.606761 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.606742 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa0e89fc-92fe-4b69-9f63-6e94c05a01e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-hwzg2\" (UID: \"aa0e89fc-92fe-4b69-9f63-6e94c05a01e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:09.614793 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.614768 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6dn6\" (UniqueName: \"kubernetes.io/projected/aa0e89fc-92fe-4b69-9f63-6e94c05a01e0-kube-api-access-r6dn6\") pod \"kuadrant-operator-controller-manager-678d87766f-hwzg2\" (UID: \"aa0e89fc-92fe-4b69-9f63-6e94c05a01e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:09.746476 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:09.746393 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:10.028327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:10.028302 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2"] Nov 02 22:35:10.031542 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:35:10.031512 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0e89fc_92fe_4b69_9f63_6e94c05a01e0.slice/crio-866185f3bf9f1fbe100a419a581e8d221d6635605933f6ca1a8efd034bcd6cd5 WatchSource:0}: Error finding container 866185f3bf9f1fbe100a419a581e8d221d6635605933f6ca1a8efd034bcd6cd5: Status 404 returned error can't find the container with id 866185f3bf9f1fbe100a419a581e8d221d6635605933f6ca1a8efd034bcd6cd5 Nov 02 22:35:10.639013 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:10.638981 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" event={"ID":"aa0e89fc-92fe-4b69-9f63-6e94c05a01e0","Type":"ContainerStarted","Data":"b9b538a619c4722cbcc4b0ab60328afc2a995f837a30155ef088de4dea9a7bb7"} Nov 02 22:35:10.639451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:10.639035 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" event={"ID":"aa0e89fc-92fe-4b69-9f63-6e94c05a01e0","Type":"ContainerStarted","Data":"866185f3bf9f1fbe100a419a581e8d221d6635605933f6ca1a8efd034bcd6cd5"} Nov 02 22:35:10.639451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:10.639073 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:10.659648 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:10.659579 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" podStartSLOduration=1.659564437 podStartE2EDuration="1.659564437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:35:10.658137265 +0000 UTC m=+2238.596250481" watchObservedRunningTime="2026-11-02 22:35:10.659564437 +0000 UTC m=+2238.597677655" Nov 02 22:35:11.595882 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:35:11.595836 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:35:21.022465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:21.022424 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb"] Nov 02 22:35:21.024418 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:21.024395 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dtnlb"] Nov 02 22:35:21.647056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:21.647027 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-hwzg2" Nov 02 22:35:22.187053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:22.187021 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a20d16-10da-4b9b-8fa5-acdcc1cc7082" path="/var/lib/kubelet/pods/22a20d16-10da-4b9b-8fa5-acdcc1cc7082/volumes" Nov 02 22:35:31.339539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.339500 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8"] Nov 02 22:35:31.343290 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.343262 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.347653 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.347627 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:35:31.348270 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.348249 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:35:31.348409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.348292 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:35:31.348409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.348320 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:35:31.351371 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.351352 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8"] Nov 02 22:35:31.365320 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.365294 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-j2rz8\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.365437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.365339 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-j2rz8\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.365437 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.365398 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fpd4\" (UniqueName: \"kubernetes.io/projected/fd72a4fb-b392-40ec-93e5-641eabdc6d80-kube-api-access-6fpd4\") pod \"maas-api-revoke-sub-d576e68b-j2rz8\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.465857 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.465828 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-j2rz8\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.466033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.465878 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-j2rz8\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.466033 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.465902 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fpd4\" (UniqueName: \"kubernetes.io/projected/fd72a4fb-b392-40ec-93e5-641eabdc6d80-kube-api-access-6fpd4\") pod \"maas-api-revoke-sub-d576e68b-j2rz8\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.466517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.466492 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-j2rz8\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.468308 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.468285 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-j2rz8\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.473511 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.473488 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fpd4\" (UniqueName: \"kubernetes.io/projected/fd72a4fb-b392-40ec-93e5-641eabdc6d80-kube-api-access-6fpd4\") pod \"maas-api-revoke-sub-d576e68b-j2rz8\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.658514 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.658442 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:31.932111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:31.932075 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8"] Nov 02 22:35:31.933448 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:35:31.933421 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd72a4fb_b392_40ec_93e5_641eabdc6d80.slice/crio-04d157eb2e7d72f4661798b31813c0cb393a20e646a9aa9bad8b2535fb2f3f9b WatchSource:0}: Error finding container 04d157eb2e7d72f4661798b31813c0cb393a20e646a9aa9bad8b2535fb2f3f9b: Status 404 returned error can't find the container with id 04d157eb2e7d72f4661798b31813c0cb393a20e646a9aa9bad8b2535fb2f3f9b Nov 02 22:35:32.781048 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:32.781010 4309 generic.go:356] "Generic (PLEG): container finished" podID="fd72a4fb-b392-40ec-93e5-641eabdc6d80" containerID="16820c38faf34f0e26d0fcff7caeb0165bc8a86779a4a6308c81cd0a69c9c033" exitCode=0 Nov 02 22:35:32.781048 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:32.781041 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" event={"ID":"fd72a4fb-b392-40ec-93e5-641eabdc6d80","Type":"ContainerDied","Data":"16820c38faf34f0e26d0fcff7caeb0165bc8a86779a4a6308c81cd0a69c9c033"} Nov 02 22:35:32.781488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:32.781090 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" event={"ID":"fd72a4fb-b392-40ec-93e5-641eabdc6d80","Type":"ContainerStarted","Data":"04d157eb2e7d72f4661798b31813c0cb393a20e646a9aa9bad8b2535fb2f3f9b"} Nov 02 22:35:33.928864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:33.928839 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:33.985404 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:33.985375 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd72a4fb-b392-40ec-93e5-641eabdc6d80-kube-api-access-6fpd4\" (UniqueName: \"kubernetes.io/projected/fd72a4fb-b392-40ec-93e5-641eabdc6d80-kube-api-access-6fpd4\") pod \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " Nov 02 22:35:33.985562 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:33.985415 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-openshift-ca\") pod \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " Nov 02 22:35:33.985562 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:33.985463 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-xks-ca\") pod \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\" (UID: \"fd72a4fb-b392-40ec-93e5-641eabdc6d80\") " Nov 02 22:35:33.985866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:33.985843 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-openshift-ca" pod "fd72a4fb-b392-40ec-93e5-641eabdc6d80" (UID: "fd72a4fb-b392-40ec-93e5-641eabdc6d80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:33.987543 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:33.987514 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd72a4fb-b392-40ec-93e5-641eabdc6d80-kube-api-access-6fpd4" pod "fd72a4fb-b392-40ec-93e5-641eabdc6d80" (UID: "fd72a4fb-b392-40ec-93e5-641eabdc6d80"). InnerVolumeSpecName "kube-api-access-6fpd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:33.987652 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:33.987611 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-xks-ca" pod "fd72a4fb-b392-40ec-93e5-641eabdc6d80" (UID: "fd72a4fb-b392-40ec-93e5-641eabdc6d80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:34.087075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:34.087047 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fpd4\" (UniqueName: \"kubernetes.io/projected/fd72a4fb-b392-40ec-93e5-641eabdc6d80-kube-api-access-6fpd4\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:34.087075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:34.087071 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:34.087075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:34.087081 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd72a4fb-b392-40ec-93e5-641eabdc6d80-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:34.794020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:34.793991 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" Nov 02 22:35:34.794190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:34.794028 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8" event={"ID":"fd72a4fb-b392-40ec-93e5-641eabdc6d80","Type":"ContainerDied","Data":"04d157eb2e7d72f4661798b31813c0cb393a20e646a9aa9bad8b2535fb2f3f9b"} Nov 02 22:35:34.794190 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:34.794077 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d157eb2e7d72f4661798b31813c0cb393a20e646a9aa9bad8b2535fb2f3f9b" Nov 02 22:35:40.019840 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:40.019793 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld"] Nov 02 22:35:40.024618 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:40.024586 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-l6jld"] Nov 02 22:35:40.187279 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:40.187251 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8" path="/var/lib/kubelet/pods/f1d0782c-f5f0-4a2f-a16f-03edcb53bcb8/volumes" Nov 02 22:35:41.595562 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:35:41.595516 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:35:46.362829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:46.362770 4309 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-696f47c6c-xlsg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 22:35:46.363258 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:46.362854 4309 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-696f47c6c-xlsg6" podUID="e9c4918d-e8b3-4e8a-a8ee-651b35456661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 22:35:50.287612 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.287576 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58"] Nov 02 22:35:50.288037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.288026 4309 cpu_manager.go:414] "containerMap: removing container" podUID="fd72a4fb-b392-40ec-93e5-641eabdc6d80" containerName="revoke-keys" Nov 02 22:35:50.288037 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.288037 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd72a4fb-b392-40ec-93e5-641eabdc6d80" containerName="revoke-keys" Nov 02 22:35:50.288122 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.288116 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd72a4fb-b392-40ec-93e5-641eabdc6d80" containerName="revoke-keys" Nov 02 22:35:50.291236 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.291220 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.296297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.296273 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:35:50.296297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.296283 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:35:50.297184 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.297160 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:35:50.297336 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.297210 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:35:50.300388 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.300364 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58"] Nov 02 22:35:50.380925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.380892 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9cd58\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.381123 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.381078 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9cd58\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.381123 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.381117 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lzf\" (UniqueName: \"kubernetes.io/projected/dfe4f25b-6b12-4871-a13b-9bb313a1548e-kube-api-access-p8lzf\") pod \"maas-api-revoke-sub-d2bc93bc-9cd58\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.482461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.482425 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9cd58\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.482461 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.482460 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lzf\" (UniqueName: \"kubernetes.io/projected/dfe4f25b-6b12-4871-a13b-9bb313a1548e-kube-api-access-p8lzf\") pod \"maas-api-revoke-sub-d2bc93bc-9cd58\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.482705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.482505 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9cd58\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.483176 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.483154 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9cd58\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.484765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.484745 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9cd58\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.489734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.489714 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lzf\" (UniqueName: \"kubernetes.io/projected/dfe4f25b-6b12-4871-a13b-9bb313a1548e-kube-api-access-p8lzf\") pod \"maas-api-revoke-sub-d2bc93bc-9cd58\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.605735 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.605708 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:50.881173 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.881148 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58"] Nov 02 22:35:50.883487 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:35:50.883454 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe4f25b_6b12_4871_a13b_9bb313a1548e.slice/crio-1d0102de9a6cf5fbf9297d52496bff588955ee8774e804ac159bfb15950d28cf WatchSource:0}: Error finding container 1d0102de9a6cf5fbf9297d52496bff588955ee8774e804ac159bfb15950d28cf: Status 404 returned error can't find the container with id 1d0102de9a6cf5fbf9297d52496bff588955ee8774e804ac159bfb15950d28cf Nov 02 22:35:50.894449 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:50.894416 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" event={"ID":"dfe4f25b-6b12-4871-a13b-9bb313a1548e","Type":"ContainerStarted","Data":"1d0102de9a6cf5fbf9297d52496bff588955ee8774e804ac159bfb15950d28cf"} Nov 02 22:35:51.901781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:51.901745 4309 generic.go:356] "Generic (PLEG): container finished" podID="dfe4f25b-6b12-4871-a13b-9bb313a1548e" containerID="a73626041e17748e028c0749eabe0a51e1dc710cefa91c34c8cd53bd31f56991" exitCode=0 Nov 02 22:35:51.902186 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:51.901830 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" event={"ID":"dfe4f25b-6b12-4871-a13b-9bb313a1548e","Type":"ContainerDied","Data":"a73626041e17748e028c0749eabe0a51e1dc710cefa91c34c8cd53bd31f56991"} Nov 02 22:35:53.049787 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.049756 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:53.209183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.209105 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe4f25b-6b12-4871-a13b-9bb313a1548e-kube-api-access-p8lzf\" (UniqueName: \"kubernetes.io/projected/dfe4f25b-6b12-4871-a13b-9bb313a1548e-kube-api-access-p8lzf\") pod \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " Nov 02 22:35:53.209183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.209171 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-openshift-ca\") pod \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " Nov 02 22:35:53.209367 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.209226 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-xks-ca\") pod \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\" (UID: \"dfe4f25b-6b12-4871-a13b-9bb313a1548e\") " Nov 02 22:35:53.209599 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.209568 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-openshift-ca" pod "dfe4f25b-6b12-4871-a13b-9bb313a1548e" (UID: "dfe4f25b-6b12-4871-a13b-9bb313a1548e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:35:53.211280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.211256 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-xks-ca" pod "dfe4f25b-6b12-4871-a13b-9bb313a1548e" (UID: "dfe4f25b-6b12-4871-a13b-9bb313a1548e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:35:53.211280 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.211268 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe4f25b-6b12-4871-a13b-9bb313a1548e-kube-api-access-p8lzf" pod "dfe4f25b-6b12-4871-a13b-9bb313a1548e" (UID: "dfe4f25b-6b12-4871-a13b-9bb313a1548e"). InnerVolumeSpecName "kube-api-access-p8lzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:35:53.310756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.310719 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8lzf\" (UniqueName: \"kubernetes.io/projected/dfe4f25b-6b12-4871-a13b-9bb313a1548e-kube-api-access-p8lzf\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:53.310756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.310746 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:53.310756 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.310757 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfe4f25b-6b12-4871-a13b-9bb313a1548e-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:35:53.913392 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.913364 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" Nov 02 22:35:53.913577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.913363 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58" event={"ID":"dfe4f25b-6b12-4871-a13b-9bb313a1548e","Type":"ContainerDied","Data":"1d0102de9a6cf5fbf9297d52496bff588955ee8774e804ac159bfb15950d28cf"} Nov 02 22:35:53.913577 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:53.913493 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d0102de9a6cf5fbf9297d52496bff588955ee8774e804ac159bfb15950d28cf" Nov 02 22:35:58.021494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:58.021451 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn"] Nov 02 22:35:58.026083 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:58.026061 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-nb7fn"] Nov 02 22:35:58.186843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:35:58.186812 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675d7445-63dc-4369-99c0-dbc3382265b9" path="/var/lib/kubelet/pods/675d7445-63dc-4369-99c0-dbc3382265b9/volumes" Nov 02 22:36:01.025785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:01.025753 4309 scope.go:122] "RemoveContainer" containerID="89108629198a05431cc20c8b1806d8cf75898f1fbc3a37726118b3a1bba2d229" Nov 02 22:36:01.036925 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:01.036903 4309 scope.go:122] "RemoveContainer" containerID="96c5472b28dc53cc721b69f88c21e5e29bd6bfa3d6bb09582def803599fe9d64" Nov 02 22:36:01.046208 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:01.046187 4309 scope.go:122] "RemoveContainer" containerID="2610d98ad3267207cf41288fc903c15b8d6d3c451972dc2a5320d49b3d845a4a" Nov 02 22:36:01.056459 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:01.056434 4309 scope.go:122] "RemoveContainer" containerID="210b771f8d3b3f4fcec054ca78d6b87dd048ab9c2c68c06633aa0f67e2ec5ebe" Nov 02 22:36:11.596004 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:36:11.595916 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:36:41.595649 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:36:41.595608 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:36:49.023759 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:49.023723 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh"] Nov 02 22:36:49.025397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:49.025374 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-f74fh"] Nov 02 22:36:50.187137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:50.187104 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85469ba4-e983-42c1-b34b-c8f761fba381" path="/var/lib/kubelet/pods/85469ba4-e983-42c1-b34b-c8f761fba381/volumes" Nov 02 22:36:53.193930 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.193896 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm"] Nov 02 22:36:53.194345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.194332 4309 cpu_manager.go:414] "containerMap: removing container" podUID="dfe4f25b-6b12-4871-a13b-9bb313a1548e" containerName="revoke-keys" Nov 02 22:36:53.194397 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.194346 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe4f25b-6b12-4871-a13b-9bb313a1548e" containerName="revoke-keys" Nov 02 22:36:53.194466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.194456 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe4f25b-6b12-4871-a13b-9bb313a1548e" containerName="revoke-keys" Nov 02 22:36:53.197958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.197941 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.203335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.203297 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:36:53.205738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.204841 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:36:53.205738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.204904 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:36:53.205738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.205255 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:36:53.206812 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.206553 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm"] Nov 02 22:36:53.315635 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.315594 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4115f8-bp7xm\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.315848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.315641 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtzt\" (UniqueName: \"kubernetes.io/projected/b6f5b07e-61d6-4657-876c-3773c5634c63-kube-api-access-hrtzt\") pod \"maas-api-revoke-sub-8d4115f8-bp7xm\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.315848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.315670 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4115f8-bp7xm\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.416623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.416586 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4115f8-bp7xm\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.416623 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.416626 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtzt\" (UniqueName: \"kubernetes.io/projected/b6f5b07e-61d6-4657-876c-3773c5634c63-kube-api-access-hrtzt\") pod \"maas-api-revoke-sub-8d4115f8-bp7xm\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.416921 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.416651 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4115f8-bp7xm\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.417314 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.417290 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d4115f8-bp7xm\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.419092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.419069 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d4115f8-bp7xm\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.424599 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.424574 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtzt\" (UniqueName: \"kubernetes.io/projected/b6f5b07e-61d6-4657-876c-3773c5634c63-kube-api-access-hrtzt\") pod \"maas-api-revoke-sub-8d4115f8-bp7xm\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.513884 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.513796 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:53.789209 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:53.789183 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm"] Nov 02 22:36:53.792301 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:36:53.792268 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f5b07e_61d6_4657_876c_3773c5634c63.slice/crio-7675ce9b4cff134655e8ed5b0e3b4ec83410af6abb7830851d40e957a95ccf8d WatchSource:0}: Error finding container 7675ce9b4cff134655e8ed5b0e3b4ec83410af6abb7830851d40e957a95ccf8d: Status 404 returned error can't find the container with id 7675ce9b4cff134655e8ed5b0e3b4ec83410af6abb7830851d40e957a95ccf8d Nov 02 22:36:54.257140 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:54.257098 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" event={"ID":"b6f5b07e-61d6-4657-876c-3773c5634c63","Type":"ContainerStarted","Data":"89ef9168bb00e1b113472dfbc3de5ceb54a836d71f5c3ca624a869b0741a8661"} Nov 02 22:36:54.257533 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:54.257167 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" event={"ID":"b6f5b07e-61d6-4657-876c-3773c5634c63","Type":"ContainerStarted","Data":"7675ce9b4cff134655e8ed5b0e3b4ec83410af6abb7830851d40e957a95ccf8d"} Nov 02 22:36:54.275454 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:54.275410 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" podStartSLOduration=1.275396809 podStartE2EDuration="1.275396809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:36:54.27261481 +0000 UTC m=+2342.210728028" watchObservedRunningTime="2026-11-02 22:36:54.275396809 +0000 UTC m=+2342.213510027" Nov 02 22:36:55.265039 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:55.265004 4309 generic.go:356] "Generic (PLEG): container finished" podID="b6f5b07e-61d6-4657-876c-3773c5634c63" containerID="89ef9168bb00e1b113472dfbc3de5ceb54a836d71f5c3ca624a869b0741a8661" exitCode=0 Nov 02 22:36:55.265536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:55.265082 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" event={"ID":"b6f5b07e-61d6-4657-876c-3773c5634c63","Type":"ContainerDied","Data":"89ef9168bb00e1b113472dfbc3de5ceb54a836d71f5c3ca624a869b0741a8661"} Nov 02 22:36:56.410068 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.410045 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:56.548260 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.548184 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f5b07e-61d6-4657-876c-3773c5634c63-kube-api-access-hrtzt\" (UniqueName: \"kubernetes.io/projected/b6f5b07e-61d6-4657-876c-3773c5634c63-kube-api-access-hrtzt\") pod \"b6f5b07e-61d6-4657-876c-3773c5634c63\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " Nov 02 22:36:56.548420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.548316 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-xks-ca\") pod \"b6f5b07e-61d6-4657-876c-3773c5634c63\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " Nov 02 22:36:56.548420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.548364 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-openshift-ca\") pod \"b6f5b07e-61d6-4657-876c-3773c5634c63\" (UID: \"b6f5b07e-61d6-4657-876c-3773c5634c63\") " Nov 02 22:36:56.548792 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.548766 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-openshift-ca" pod "b6f5b07e-61d6-4657-876c-3773c5634c63" (UID: "b6f5b07e-61d6-4657-876c-3773c5634c63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:36:56.550298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.550276 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-xks-ca" pod "b6f5b07e-61d6-4657-876c-3773c5634c63" (UID: "b6f5b07e-61d6-4657-876c-3773c5634c63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:36:56.550381 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.550296 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f5b07e-61d6-4657-876c-3773c5634c63-kube-api-access-hrtzt" pod "b6f5b07e-61d6-4657-876c-3773c5634c63" (UID: "b6f5b07e-61d6-4657-876c-3773c5634c63"). InnerVolumeSpecName "kube-api-access-hrtzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:36:56.649409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.649375 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:56.649409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.649405 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrtzt\" (UniqueName: \"kubernetes.io/projected/b6f5b07e-61d6-4657-876c-3773c5634c63-kube-api-access-hrtzt\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:56.649409 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:56.649415 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f5b07e-61d6-4657-876c-3773c5634c63-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:36:57.278566 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:57.278537 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" Nov 02 22:36:57.278566 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:57.278559 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm" event={"ID":"b6f5b07e-61d6-4657-876c-3773c5634c63","Type":"ContainerDied","Data":"7675ce9b4cff134655e8ed5b0e3b4ec83410af6abb7830851d40e957a95ccf8d"} Nov 02 22:36:57.278804 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:57.278615 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7675ce9b4cff134655e8ed5b0e3b4ec83410af6abb7830851d40e957a95ccf8d" Nov 02 22:36:59.027719 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:59.027662 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x"] Nov 02 22:36:59.029620 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:36:59.029598 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-vk29x"] Nov 02 22:37:00.187129 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:00.187095 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f99e21c-cf67-420c-b738-485cf00ba3db" path="/var/lib/kubelet/pods/1f99e21c-cf67-420c-b738-485cf00ba3db/volumes" Nov 02 22:37:01.088498 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:01.088464 4309 scope.go:122] "RemoveContainer" containerID="afd88bb4960e9de7098c2b1144d298c881a2ec4352c40e50577109a7f65bb3e9" Nov 02 22:37:01.097670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:01.097650 4309 scope.go:122] "RemoveContainer" containerID="c17f4a0ab97ccd5d432913df4de2623ecbd3a0be1890c55a36e7024004d2ae40" Nov 02 22:37:01.106196 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:01.106176 4309 scope.go:122] "RemoveContainer" containerID="c71d96b0b9f8a99716e0e6550a8b50f4026c7e1fcd621eb15261651459682a4a" Nov 02 22:37:04.117751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.117707 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz"] Nov 02 22:37:04.118505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.118474 4309 cpu_manager.go:414] "containerMap: removing container" podUID="b6f5b07e-61d6-4657-876c-3773c5634c63" containerName="revoke-keys" Nov 02 22:37:04.118505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.118500 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f5b07e-61d6-4657-876c-3773c5634c63" containerName="revoke-keys" Nov 02 22:37:04.118713 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.118630 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f5b07e-61d6-4657-876c-3773c5634c63" containerName="revoke-keys" Nov 02 22:37:04.124458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.124438 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.126942 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.126915 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz"] Nov 02 22:37:04.128997 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.128975 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:37:04.129110 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.129011 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:37:04.129110 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.129058 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:37:04.129249 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.129124 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:37:04.224943 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.224910 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvq5h\" (UniqueName: \"kubernetes.io/projected/7d08f27b-9c70-4e67-b244-1357688488a7-kube-api-access-mvq5h\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.225115 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.225056 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.225115 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.225107 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.326254 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.326217 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.326465 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.326449 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.326642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.326621 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvq5h\" (UniqueName: \"kubernetes.io/projected/7d08f27b-9c70-4e67-b244-1357688488a7-kube-api-access-mvq5h\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.327103 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.327083 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.329018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.328999 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.333829 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.333808 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvq5h\" (UniqueName: \"kubernetes.io/projected/7d08f27b-9c70-4e67-b244-1357688488a7-kube-api-access-mvq5h\") pod \"maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.442768 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.442686 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:04.720401 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:04.720374 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz"] Nov 02 22:37:04.722506 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:37:04.722465 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d08f27b_9c70_4e67_b244_1357688488a7.slice/crio-0460e9ac643568e9a4911989eb89c0816456f7eded222ac6a90c051745042bf0 WatchSource:0}: Error finding container 0460e9ac643568e9a4911989eb89c0816456f7eded222ac6a90c051745042bf0: Status 404 returned error can't find the container with id 0460e9ac643568e9a4911989eb89c0816456f7eded222ac6a90c051745042bf0 Nov 02 22:37:05.326296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:05.326259 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" event={"ID":"7d08f27b-9c70-4e67-b244-1357688488a7","Type":"ContainerStarted","Data":"71cbb6976151adea753d154e1cc953256d4e462becbc86bb8131bf50d91d5ff0"} Nov 02 22:37:05.326802 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:05.326325 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" event={"ID":"7d08f27b-9c70-4e67-b244-1357688488a7","Type":"ContainerStarted","Data":"0460e9ac643568e9a4911989eb89c0816456f7eded222ac6a90c051745042bf0"} Nov 02 22:37:05.343243 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:05.343197 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" podStartSLOduration=1.343183317 podStartE2EDuration="1.343183317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:37:05.342730776 +0000 UTC m=+2353.280843994" watchObservedRunningTime="2026-11-02 22:37:05.343183317 +0000 UTC m=+2353.281296535" Nov 02 22:37:06.333642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:06.333604 4309 generic.go:356] "Generic (PLEG): container finished" podID="7d08f27b-9c70-4e67-b244-1357688488a7" containerID="71cbb6976151adea753d154e1cc953256d4e462becbc86bb8131bf50d91d5ff0" exitCode=0 Nov 02 22:37:06.334053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:06.333697 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" event={"ID":"7d08f27b-9c70-4e67-b244-1357688488a7","Type":"ContainerDied","Data":"71cbb6976151adea753d154e1cc953256d4e462becbc86bb8131bf50d91d5ff0"} Nov 02 22:37:07.489224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.489195 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:07.664174 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.664076 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d08f27b-9c70-4e67-b244-1357688488a7-kube-api-access-mvq5h\" (UniqueName: \"kubernetes.io/projected/7d08f27b-9c70-4e67-b244-1357688488a7-kube-api-access-mvq5h\") pod \"7d08f27b-9c70-4e67-b244-1357688488a7\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " Nov 02 22:37:07.664354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.664191 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-openshift-ca\") pod \"7d08f27b-9c70-4e67-b244-1357688488a7\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " Nov 02 22:37:07.664354 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.664310 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-xks-ca\") pod \"7d08f27b-9c70-4e67-b244-1357688488a7\" (UID: \"7d08f27b-9c70-4e67-b244-1357688488a7\") " Nov 02 22:37:07.664652 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.664609 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-openshift-ca" pod "7d08f27b-9c70-4e67-b244-1357688488a7" (UID: "7d08f27b-9c70-4e67-b244-1357688488a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:37:07.666294 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.666266 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d08f27b-9c70-4e67-b244-1357688488a7-kube-api-access-mvq5h" pod "7d08f27b-9c70-4e67-b244-1357688488a7" (UID: "7d08f27b-9c70-4e67-b244-1357688488a7"). InnerVolumeSpecName "kube-api-access-mvq5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:37:07.667065 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.667038 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-xks-ca" pod "7d08f27b-9c70-4e67-b244-1357688488a7" (UID: "7d08f27b-9c70-4e67-b244-1357688488a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:37:07.765783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.765749 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:07.765783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.765778 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvq5h\" (UniqueName: \"kubernetes.io/projected/7d08f27b-9c70-4e67-b244-1357688488a7-kube-api-access-mvq5h\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:07.765783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:07.765791 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d08f27b-9c70-4e67-b244-1357688488a7-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:37:08.345345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:08.345310 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" event={"ID":"7d08f27b-9c70-4e67-b244-1357688488a7","Type":"ContainerDied","Data":"0460e9ac643568e9a4911989eb89c0816456f7eded222ac6a90c051745042bf0"} Nov 02 22:37:08.345553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:08.345368 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0460e9ac643568e9a4911989eb89c0816456f7eded222ac6a90c051745042bf0" Nov 02 22:37:08.345553 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:08.345325 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz" Nov 02 22:37:11.597284 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:37:11.597246 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:37:14.769504 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:14.769449 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz"] Nov 02 22:37:14.771406 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:14.771382 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-93446a-wrdqz"] Nov 02 22:37:16.188486 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:16.188443 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d08f27b-9c70-4e67-b244-1357688488a7" path="/var/lib/kubelet/pods/7d08f27b-9c70-4e67-b244-1357688488a7/volumes" Nov 02 22:37:41.594536 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:37:41.594454 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:37:56.597405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:56.597373 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:37:56.609722 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:56.609703 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:37:59.012481 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.012438 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7"] Nov 02 22:37:59.012933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.012905 4309 cpu_manager.go:414] "containerMap: removing container" podUID="7d08f27b-9c70-4e67-b244-1357688488a7" containerName="revoke-keys" Nov 02 22:37:59.012933 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.012918 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d08f27b-9c70-4e67-b244-1357688488a7" containerName="revoke-keys" Nov 02 22:37:59.013020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.012988 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d08f27b-9c70-4e67-b244-1357688488a7" containerName="revoke-keys" Nov 02 22:37:59.016338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.016322 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.021453 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.021431 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:37:59.025890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.021811 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:37:59.025890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.021825 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:37:59.025890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.021851 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:37:59.025890 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.024583 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7"] Nov 02 22:37:59.062101 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.062061 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6v4d7\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.062272 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.062204 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4f69\" (UniqueName: \"kubernetes.io/projected/0bf06653-f7ea-406a-86f2-c41aa9f59d28-kube-api-access-d4f69\") pod \"maas-api-revoke-sub-452b6d7b-6v4d7\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.062328 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.062314 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6v4d7\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.163079 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.163045 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6v4d7\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.163297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.163109 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6v4d7\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.163297 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.163161 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f69\" (UniqueName: \"kubernetes.io/projected/0bf06653-f7ea-406a-86f2-c41aa9f59d28-kube-api-access-d4f69\") pod \"maas-api-revoke-sub-452b6d7b-6v4d7\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.163929 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.163900 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-6v4d7\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.165483 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.165454 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-6v4d7\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.170879 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.170854 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4f69\" (UniqueName: \"kubernetes.io/projected/0bf06653-f7ea-406a-86f2-c41aa9f59d28-kube-api-access-d4f69\") pod \"maas-api-revoke-sub-452b6d7b-6v4d7\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.333007 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.332972 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:37:59.610551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.610522 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7"] Nov 02 22:37:59.611897 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:37:59.611863 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf06653_f7ea_406a_86f2_c41aa9f59d28.slice/crio-770c553afbc4ff61a784855d96d65eefd55b559092ca9abf3f8f1ca97f492400 WatchSource:0}: Error finding container 770c553afbc4ff61a784855d96d65eefd55b559092ca9abf3f8f1ca97f492400: Status 404 returned error can't find the container with id 770c553afbc4ff61a784855d96d65eefd55b559092ca9abf3f8f1ca97f492400 Nov 02 22:37:59.640130 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:37:59.640102 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" event={"ID":"0bf06653-f7ea-406a-86f2-c41aa9f59d28","Type":"ContainerStarted","Data":"770c553afbc4ff61a784855d96d65eefd55b559092ca9abf3f8f1ca97f492400"} Nov 02 22:38:00.650645 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:00.650598 4309 generic.go:356] "Generic (PLEG): container finished" podID="0bf06653-f7ea-406a-86f2-c41aa9f59d28" containerID="c241d481813f57958b3e81d25199cabaaacee8983b71500688f032a57af24e24" exitCode=0 Nov 02 22:38:00.651150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:00.650717 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" event={"ID":"0bf06653-f7ea-406a-86f2-c41aa9f59d28","Type":"ContainerDied","Data":"c241d481813f57958b3e81d25199cabaaacee8983b71500688f032a57af24e24"} Nov 02 22:38:01.136458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:01.136415 4309 scope.go:122] "RemoveContainer" containerID="306c025135f9161321729e5337e64d67b0f12b4a820502dee8ba4f7349302dd8" Nov 02 22:38:01.153897 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:01.153847 4309 scope.go:122] "RemoveContainer" containerID="006988607ef8a3e1365f2b8e0e2e8d1df68d61d443842ceaba409a68ace2edf2" Nov 02 22:38:01.166977 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:01.166881 4309 scope.go:122] "RemoveContainer" containerID="a64e119ac81510d870b5bb1064278075f0717f78983ea66a823ff52d36a00f1d" Nov 02 22:38:01.225958 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:01.215213 4309 scope.go:122] "RemoveContainer" containerID="5c687f3d287eba8d971d6a9d99c0a48f8731abb730d3a61f593efc70867ea910" Nov 02 22:38:02.161702 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.160770 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:38:02.264075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.261007 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-xks-ca\") pod \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " Nov 02 22:38:02.264075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.261139 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-openshift-ca\") pod \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " Nov 02 22:38:02.264075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.261237 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf06653-f7ea-406a-86f2-c41aa9f59d28-kube-api-access-d4f69\" (UniqueName: \"kubernetes.io/projected/0bf06653-f7ea-406a-86f2-c41aa9f59d28-kube-api-access-d4f69\") pod \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\" (UID: \"0bf06653-f7ea-406a-86f2-c41aa9f59d28\") " Nov 02 22:38:02.264075 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.264023 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-openshift-ca" pod "0bf06653-f7ea-406a-86f2-c41aa9f59d28" (UID: "0bf06653-f7ea-406a-86f2-c41aa9f59d28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:02.267684 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.266380 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf06653-f7ea-406a-86f2-c41aa9f59d28-kube-api-access-d4f69" pod "0bf06653-f7ea-406a-86f2-c41aa9f59d28" (UID: "0bf06653-f7ea-406a-86f2-c41aa9f59d28"). InnerVolumeSpecName "kube-api-access-d4f69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:02.268170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.268135 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-xks-ca" pod "0bf06653-f7ea-406a-86f2-c41aa9f59d28" (UID: "0bf06653-f7ea-406a-86f2-c41aa9f59d28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:02.362658 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.362539 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:02.366825 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.366792 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bf06653-f7ea-406a-86f2-c41aa9f59d28-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:02.367043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.367024 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4f69\" (UniqueName: \"kubernetes.io/projected/0bf06653-f7ea-406a-86f2-c41aa9f59d28-kube-api-access-d4f69\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:02.672187 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.672095 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" Nov 02 22:38:02.672501 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.672121 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7" event={"ID":"0bf06653-f7ea-406a-86f2-c41aa9f59d28","Type":"ContainerDied","Data":"770c553afbc4ff61a784855d96d65eefd55b559092ca9abf3f8f1ca97f492400"} Nov 02 22:38:02.672604 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:02.672539 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="770c553afbc4ff61a784855d96d65eefd55b559092ca9abf3f8f1ca97f492400" Nov 02 22:38:11.622415 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:38:11.622373 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:38:14.795584 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.795548 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg"] Nov 02 22:38:14.796188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.796171 4309 cpu_manager.go:414] "containerMap: removing container" podUID="0bf06653-f7ea-406a-86f2-c41aa9f59d28" containerName="revoke-keys" Nov 02 22:38:14.796188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.796189 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf06653-f7ea-406a-86f2-c41aa9f59d28" containerName="revoke-keys" Nov 02 22:38:14.796296 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.796282 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf06653-f7ea-406a-86f2-c41aa9f59d28" containerName="revoke-keys" Nov 02 22:38:14.800117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.800087 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:14.804224 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.803704 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:38:14.804914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.804894 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:38:14.805050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.804914 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:38:14.805050 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.805000 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:38:14.805765 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.805741 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg"] Nov 02 22:38:14.948561 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.948509 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9mzsg\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:14.948783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.948588 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8dfh\" (UniqueName: \"kubernetes.io/projected/3bfb2ad2-b822-494c-ab26-7766b233ab26-kube-api-access-c8dfh\") pod \"maas-api-revoke-sub-3e31a506-9mzsg\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:14.948783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:14.948732 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9mzsg\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:15.050191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.050073 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9mzsg\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:15.050191 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.050152 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8dfh\" (UniqueName: \"kubernetes.io/projected/3bfb2ad2-b822-494c-ab26-7766b233ab26-kube-api-access-c8dfh\") pod \"maas-api-revoke-sub-3e31a506-9mzsg\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:15.050466 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.050202 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9mzsg\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:15.051137 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.051109 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-9mzsg\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:15.053979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.053948 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-9mzsg\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:15.058632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.058599 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8dfh\" (UniqueName: \"kubernetes.io/projected/3bfb2ad2-b822-494c-ab26-7766b233ab26-kube-api-access-c8dfh\") pod \"maas-api-revoke-sub-3e31a506-9mzsg\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:15.118494 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.118454 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:15.416771 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.416735 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg"] Nov 02 22:38:15.428819 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:38:15.419121 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfb2ad2_b822_494c_ab26_7766b233ab26.slice/crio-ae050457ab4f5caf70d887ee833daf18e21419bf963ed8241c4228a2b22363ab WatchSource:0}: Error finding container ae050457ab4f5caf70d887ee833daf18e21419bf963ed8241c4228a2b22363ab: Status 404 returned error can't find the container with id ae050457ab4f5caf70d887ee833daf18e21419bf963ed8241c4228a2b22363ab Nov 02 22:38:15.776913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.776874 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" event={"ID":"3bfb2ad2-b822-494c-ab26-7766b233ab26","Type":"ContainerStarted","Data":"71150a8c68359c256bcd011a19268620171598e7058d2591e28c4e11e82231b3"} Nov 02 22:38:15.777118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.776936 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" event={"ID":"3bfb2ad2-b822-494c-ab26-7766b233ab26","Type":"ContainerStarted","Data":"ae050457ab4f5caf70d887ee833daf18e21419bf963ed8241c4228a2b22363ab"} Nov 02 22:38:15.793276 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:15.793204 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" podStartSLOduration=1.79318412 podStartE2EDuration="1.79318412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:15.792151893 +0000 UTC m=+2423.730265112" watchObservedRunningTime="2026-11-02 22:38:15.79318412 +0000 UTC m=+2423.731297339" Nov 02 22:38:16.834699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:16.816553 4309 generic.go:356] "Generic (PLEG): container finished" podID="3bfb2ad2-b822-494c-ab26-7766b233ab26" containerID="71150a8c68359c256bcd011a19268620171598e7058d2591e28c4e11e82231b3" exitCode=0 Nov 02 22:38:16.834699 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:16.816710 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" event={"ID":"3bfb2ad2-b822-494c-ab26-7766b233ab26","Type":"ContainerDied","Data":"71150a8c68359c256bcd011a19268620171598e7058d2591e28c4e11e82231b3"} Nov 02 22:38:17.973639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:17.973616 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:18.097982 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.097899 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-xks-ca\") pod \"3bfb2ad2-b822-494c-ab26-7766b233ab26\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " Nov 02 22:38:18.098150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.097999 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bfb2ad2-b822-494c-ab26-7766b233ab26-kube-api-access-c8dfh\" (UniqueName: \"kubernetes.io/projected/3bfb2ad2-b822-494c-ab26-7766b233ab26-kube-api-access-c8dfh\") pod \"3bfb2ad2-b822-494c-ab26-7766b233ab26\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " Nov 02 22:38:18.098150 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.098058 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-openshift-ca\") pod \"3bfb2ad2-b822-494c-ab26-7766b233ab26\" (UID: \"3bfb2ad2-b822-494c-ab26-7766b233ab26\") " Nov 02 22:38:18.098441 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.098411 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-openshift-ca" pod "3bfb2ad2-b822-494c-ab26-7766b233ab26" (UID: "3bfb2ad2-b822-494c-ab26-7766b233ab26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:18.100183 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.100153 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-xks-ca" pod "3bfb2ad2-b822-494c-ab26-7766b233ab26" (UID: "3bfb2ad2-b822-494c-ab26-7766b233ab26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:18.100298 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.100241 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bfb2ad2-b822-494c-ab26-7766b233ab26-kube-api-access-c8dfh" pod "3bfb2ad2-b822-494c-ab26-7766b233ab26" (UID: "3bfb2ad2-b822-494c-ab26-7766b233ab26"). InnerVolumeSpecName "kube-api-access-c8dfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:18.199457 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.199423 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:18.199457 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.199460 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8dfh\" (UniqueName: \"kubernetes.io/projected/3bfb2ad2-b822-494c-ab26-7766b233ab26-kube-api-access-c8dfh\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:18.199642 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.199477 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bfb2ad2-b822-494c-ab26-7766b233ab26-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:18.830048 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.830021 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" Nov 02 22:38:18.830231 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.830020 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg" event={"ID":"3bfb2ad2-b822-494c-ab26-7766b233ab26","Type":"ContainerDied","Data":"ae050457ab4f5caf70d887ee833daf18e21419bf963ed8241c4228a2b22363ab"} Nov 02 22:38:18.830231 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:18.830156 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae050457ab4f5caf70d887ee833daf18e21419bf963ed8241c4228a2b22363ab" Nov 02 22:38:24.020783 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:24.020743 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj"] Nov 02 22:38:24.021564 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:24.021538 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51179b85-hbftj"] Nov 02 22:38:24.187134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:24.187099 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf653ce-c37f-4098-8d11-79c9f16d0545" path="/var/lib/kubelet/pods/dcf653ce-c37f-4098-8d11-79c9f16d0545/volumes" Nov 02 22:38:35.126540 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.126502 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl"] Nov 02 22:38:35.127068 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.127022 4309 cpu_manager.go:414] "containerMap: removing container" podUID="3bfb2ad2-b822-494c-ab26-7766b233ab26" containerName="revoke-keys" Nov 02 22:38:35.127068 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.127036 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bfb2ad2-b822-494c-ab26-7766b233ab26" containerName="revoke-keys" Nov 02 22:38:35.127144 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.127096 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bfb2ad2-b822-494c-ab26-7766b233ab26" containerName="revoke-keys" Nov 02 22:38:35.130546 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.130529 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.134491 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.134463 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:38:35.135633 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.135611 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:38:35.135795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.135654 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:38:35.135795 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.135785 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:38:35.139048 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.138536 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl"] Nov 02 22:38:35.223379 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.223350 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da9d302-p2lvl\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.223521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.223388 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da9d302-p2lvl\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.223521 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.223491 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5xm\" (UniqueName: \"kubernetes.io/projected/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-kube-api-access-fn5xm\") pod \"maas-api-revoke-sub-0da9d302-p2lvl\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.323998 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.323965 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5xm\" (UniqueName: \"kubernetes.io/projected/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-kube-api-access-fn5xm\") pod \"maas-api-revoke-sub-0da9d302-p2lvl\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.324163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.324058 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da9d302-p2lvl\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.324163 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.324086 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da9d302-p2lvl\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.324692 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.324661 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da9d302-p2lvl\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.326477 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.326448 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da9d302-p2lvl\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.332010 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.331987 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5xm\" (UniqueName: \"kubernetes.io/projected/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-kube-api-access-fn5xm\") pod \"maas-api-revoke-sub-0da9d302-p2lvl\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.445306 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.445236 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:35.716747 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.716723 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl"] Nov 02 22:38:35.719067 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:38:35.719036 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cbfcb2_5a29_4cb8_a648_aefc752ef61d.slice/crio-0b4ebd99973b40205413ca088e4a388bb54cdfae9b6906c396997bdbcb465593 WatchSource:0}: Error finding container 0b4ebd99973b40205413ca088e4a388bb54cdfae9b6906c396997bdbcb465593: Status 404 returned error can't find the container with id 0b4ebd99973b40205413ca088e4a388bb54cdfae9b6906c396997bdbcb465593 Nov 02 22:38:35.949981 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:35.949954 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" event={"ID":"78cbfcb2-5a29-4cb8-a648-aefc752ef61d","Type":"ContainerStarted","Data":"0b4ebd99973b40205413ca088e4a388bb54cdfae9b6906c396997bdbcb465593"} Nov 02 22:38:36.956445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:36.956407 4309 generic.go:356] "Generic (PLEG): container finished" podID="78cbfcb2-5a29-4cb8-a648-aefc752ef61d" containerID="1b3294828ed5d6e963c85e531e2b189f68f3e54b279eb9b68ac26f7606faad32" exitCode=0 Nov 02 22:38:36.956876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:36.956496 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" event={"ID":"78cbfcb2-5a29-4cb8-a648-aefc752ef61d","Type":"ContainerDied","Data":"1b3294828ed5d6e963c85e531e2b189f68f3e54b279eb9b68ac26f7606faad32"} Nov 02 22:38:38.102531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.102501 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:38.253914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.253834 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-kube-api-access-fn5xm\" (UniqueName: \"kubernetes.io/projected/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-kube-api-access-fn5xm\") pod \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " Nov 02 22:38:38.253914 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.253878 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-xks-ca\") pod \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " Nov 02 22:38:38.254108 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.253935 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-openshift-ca\") pod \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\" (UID: \"78cbfcb2-5a29-4cb8-a648-aefc752ef61d\") " Nov 02 22:38:38.254346 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.254323 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-openshift-ca" pod "78cbfcb2-5a29-4cb8-a648-aefc752ef61d" (UID: "78cbfcb2-5a29-4cb8-a648-aefc752ef61d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:38.255889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.255865 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-kube-api-access-fn5xm" pod "78cbfcb2-5a29-4cb8-a648-aefc752ef61d" (UID: "78cbfcb2-5a29-4cb8-a648-aefc752ef61d"). InnerVolumeSpecName "kube-api-access-fn5xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:38.255981 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.255902 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-xks-ca" pod "78cbfcb2-5a29-4cb8-a648-aefc752ef61d" (UID: "78cbfcb2-5a29-4cb8-a648-aefc752ef61d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:38.355698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.355645 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5xm\" (UniqueName: \"kubernetes.io/projected/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-kube-api-access-fn5xm\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:38.355698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.355688 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:38.355698 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.355699 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cbfcb2-5a29-4cb8-a648-aefc752ef61d-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:38.968003 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.967972 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" Nov 02 22:38:38.968166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.968008 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl" event={"ID":"78cbfcb2-5a29-4cb8-a648-aefc752ef61d","Type":"ContainerDied","Data":"0b4ebd99973b40205413ca088e4a388bb54cdfae9b6906c396997bdbcb465593"} Nov 02 22:38:38.968166 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:38.968056 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b4ebd99973b40205413ca088e4a388bb54cdfae9b6906c396997bdbcb465593" Nov 02 22:38:41.595111 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:38:41.595075 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:38:47.313910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.313873 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd"] Nov 02 22:38:47.314439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.314345 4309 cpu_manager.go:414] "containerMap: removing container" podUID="78cbfcb2-5a29-4cb8-a648-aefc752ef61d" containerName="revoke-keys" Nov 02 22:38:47.314439 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.314358 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cbfcb2-5a29-4cb8-a648-aefc752ef61d" containerName="revoke-keys" Nov 02 22:38:47.314581 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.314451 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cbfcb2-5a29-4cb8-a648-aefc752ef61d" containerName="revoke-keys" Nov 02 22:38:47.317950 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.317930 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.328047 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.327482 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:38:47.329458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.328481 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd"] Nov 02 22:38:47.329458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.328822 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tcmbk\"" Nov 02 22:38:47.329458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.328867 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:38:47.329458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.328947 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 22:38:47.329458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.329061 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 22:38:47.443591 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.443548 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrkb\" (UniqueName: \"kubernetes.io/projected/c9c92f15-528d-48b5-9434-70c3cd3adb0c-kube-api-access-5zrkb\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.443781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.443692 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-metrics-certs\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.443781 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.443729 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c92f15-528d-48b5-9434-70c3cd3adb0c-openshift-service-ca\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.443960 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.443933 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-maas-api-tls\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.545338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.545303 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-metrics-certs\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.545338 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.545341 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c92f15-528d-48b5-9434-70c3cd3adb0c-openshift-service-ca\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.545588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.545419 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-maas-api-tls\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.545588 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.545447 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrkb\" (UniqueName: \"kubernetes.io/projected/c9c92f15-528d-48b5-9434-70c3cd3adb0c-kube-api-access-5zrkb\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.546147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.546117 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c92f15-528d-48b5-9434-70c3cd3adb0c-openshift-service-ca\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.547751 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.547722 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-metrics-certs\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.547877 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.547848 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-maas-api-tls\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.552968 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.552948 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrkb\" (UniqueName: \"kubernetes.io/projected/c9c92f15-528d-48b5-9434-70c3cd3adb0c-kube-api-access-5zrkb\") pod \"maas-api-746984ddc4-mmqnd\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.640854 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.640768 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:47.919149 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.919108 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd"] Nov 02 22:38:47.921088 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:38:47.921060 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c92f15_528d_48b5_9434_70c3cd3adb0c.slice/crio-fbb002940fe80a4187be6a88c9dff171f4d421e1902eb22566f6e6e6e6496233 WatchSource:0}: Error finding container fbb002940fe80a4187be6a88c9dff171f4d421e1902eb22566f6e6e6e6496233: Status 404 returned error can't find the container with id fbb002940fe80a4187be6a88c9dff171f4d421e1902eb22566f6e6e6e6496233 Nov 02 22:38:47.922749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:47.922729 4309 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 22:38:48.020991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:48.020955 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" event={"ID":"c9c92f15-528d-48b5-9434-70c3cd3adb0c","Type":"ContainerStarted","Data":"fbb002940fe80a4187be6a88c9dff171f4d421e1902eb22566f6e6e6e6496233"} Nov 02 22:38:51.041847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:51.041809 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" event={"ID":"c9c92f15-528d-48b5-9434-70c3cd3adb0c","Type":"ContainerStarted","Data":"36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557"} Nov 02 22:38:51.042324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:51.041866 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:51.059118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:51.059072 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" podStartSLOduration=1.588128897 podStartE2EDuration="4.059059341s" totalImagesPullingTime="2.470930444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:38:51.058410178 +0000 UTC m=+2458.996523396" watchObservedRunningTime="2026-11-02 22:38:51.059059341 +0000 UTC m=+2458.997172559" Nov 02 22:38:53.019313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:53.019272 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl"] Nov 02 22:38:53.022710 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:53.022669 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4ljxl"] Nov 02 22:38:54.186785 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:54.186746 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64261281-ef0a-4c2e-a9d5-7c9294e61003" path="/var/lib/kubelet/pods/64261281-ef0a-4c2e-a9d5-7c9294e61003/volumes" Nov 02 22:38:54.972993 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:54.972953 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd"] Nov 02 22:38:54.973199 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:54.973144 4309 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" podUID="c9c92f15-528d-48b5-9434-70c3cd3adb0c" containerName="maas-api" containerID="cri-o://36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557" gracePeriod=30 Nov 02 22:38:55.230773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.230709 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:55.319705 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.319642 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c92f15-528d-48b5-9434-70c3cd3adb0c-kube-api-access-5zrkb\" (UniqueName: \"kubernetes.io/projected/c9c92f15-528d-48b5-9434-70c3cd3adb0c-kube-api-access-5zrkb\") pod \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " Nov 02 22:38:55.319882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.319736 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-metrics-certs\") pod \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " Nov 02 22:38:55.319882 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.319836 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-maas-api-tls\") pod \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " Nov 02 22:38:55.319979 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.319886 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c92f15-528d-48b5-9434-70c3cd3adb0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c92f15-528d-48b5-9434-70c3cd3adb0c-openshift-service-ca\") pod \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\" (UID: \"c9c92f15-528d-48b5-9434-70c3cd3adb0c\") " Nov 02 22:38:55.320396 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.320366 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c92f15-528d-48b5-9434-70c3cd3adb0c-openshift-service-ca" pod "c9c92f15-528d-48b5-9434-70c3cd3adb0c" (UID: "c9c92f15-528d-48b5-9434-70c3cd3adb0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:38:55.321978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.321950 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-metrics-certs" pod "c9c92f15-528d-48b5-9434-70c3cd3adb0c" (UID: "c9c92f15-528d-48b5-9434-70c3cd3adb0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:55.322113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.322036 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c92f15-528d-48b5-9434-70c3cd3adb0c-kube-api-access-5zrkb" pod "c9c92f15-528d-48b5-9434-70c3cd3adb0c" (UID: "c9c92f15-528d-48b5-9434-70c3cd3adb0c"). InnerVolumeSpecName "kube-api-access-5zrkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:38:55.322248 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.322233 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-maas-api-tls" pod "c9c92f15-528d-48b5-9434-70c3cd3adb0c" (UID: "c9c92f15-528d-48b5-9434-70c3cd3adb0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:38:55.421315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.421281 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrkb\" (UniqueName: \"kubernetes.io/projected/c9c92f15-528d-48b5-9434-70c3cd3adb0c-kube-api-access-5zrkb\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:55.421315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.421312 4309 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-metrics-certs\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:55.421315 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.421322 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9c92f15-528d-48b5-9434-70c3cd3adb0c-maas-api-tls\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:55.421559 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:55.421336 4309 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9c92f15-528d-48b5-9434-70c3cd3adb0c-openshift-service-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:38:56.069001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.068965 4309 generic.go:356] "Generic (PLEG): container finished" podID="c9c92f15-528d-48b5-9434-70c3cd3adb0c" containerID="36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557" exitCode=0 Nov 02 22:38:56.069001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.069003 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" Nov 02 22:38:56.069330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.069050 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" event={"ID":"c9c92f15-528d-48b5-9434-70c3cd3adb0c","Type":"ContainerDied","Data":"36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557"} Nov 02 22:38:56.069330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.069106 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd" event={"ID":"c9c92f15-528d-48b5-9434-70c3cd3adb0c","Type":"ContainerDied","Data":"fbb002940fe80a4187be6a88c9dff171f4d421e1902eb22566f6e6e6e6496233"} Nov 02 22:38:56.069330 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.069131 4309 scope.go:122] "RemoveContainer" containerID="36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557" Nov 02 22:38:56.079534 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.079513 4309 scope.go:122] "RemoveContainer" containerID="36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557" Nov 02 22:38:56.079859 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:38:56.079835 4309 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557\": container with ID starting with 36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557 not found: ID does not exist" containerID="36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557" Nov 02 22:38:56.079924 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.079870 4309 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557"} err="failed to get container status \"36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557\": rpc error: code = NotFound desc = could not find container \"36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557\": container with ID starting with 36dfb83f2f14ca5dfb15d149e20d586b8e98017b3e4193e0e6ffdd0492cda557 not found: ID does not exist" Nov 02 22:38:56.091446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.091410 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd"] Nov 02 22:38:56.099080 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.099045 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-746984ddc4-mmqnd"] Nov 02 22:38:56.187447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:38:56.187415 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c92f15-528d-48b5-9434-70c3cd3adb0c" path="/var/lib/kubelet/pods/c9c92f15-528d-48b5-9434-70c3cd3adb0c/volumes" Nov 02 22:39:01.360800 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:01.360772 4309 scope.go:122] "RemoveContainer" containerID="affa8fa77e053d8dff760da68e18630345b482e323ded74fa2a5ea794901a654" Nov 02 22:39:11.595815 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:39:11.595727 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:39:13.290095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.290057 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh"] Nov 02 22:39:13.290821 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.290798 4309 cpu_manager.go:414] "containerMap: removing container" podUID="c9c92f15-528d-48b5-9434-70c3cd3adb0c" containerName="maas-api" Nov 02 22:39:13.290928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.290821 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c92f15-528d-48b5-9434-70c3cd3adb0c" containerName="maas-api" Nov 02 22:39:13.290928 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.290923 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c92f15-528d-48b5-9434-70c3cd3adb0c" containerName="maas-api" Nov 02 22:39:13.295752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.295731 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:13.299291 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.299268 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n7t66\"" Nov 02 22:39:13.301888 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.301862 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh"] Nov 02 22:39:13.302012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.301893 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:39:13.302012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.301903 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:39:13.302012 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.301960 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:39:13.412995 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.412950 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4hj\" (UniqueName: \"kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:13.413195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.413026 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:13.413195 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.413090 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:13.514396 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.514357 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4hj\" (UniqueName: \"kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:13.514609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.514420 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:13.514609 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.514442 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:13.515139 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.515117 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:13.516745 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:13.516724 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:13.520202 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:39:13.520185 4309 projected.go:196] Error preparing data for projected volume kube-api-access-fq4hj for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 22:39:13.520300 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:39:13.520244 4309 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj podName:72ee8e51-3153-4acc-b025-c964a52be17d nodeName:}" failed. No retries permitted until 2026-11-02 22:39:14.020229294 +0000 UTC m=+2481.958342506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fq4hj" (UniqueName: "kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj") pod "maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" (UID: "72ee8e51-3153-4acc-b025-c964a52be17d") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 02 22:39:14.020711 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:14.020658 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4hj\" (UniqueName: \"kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:14.028605 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:14.028583 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4hj\" (UniqueName: \"kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj\") pod \"maas-api-revoke-keys-e2e-ait-555ba599-qh6fh\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:14.213469 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:14.213440 4309 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" secret="" err="secret \"maas-api-cleanup-dockercfg-n7t66\" not found" Nov 02 22:39:14.213651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:14.213501 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:14.488056 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:14.488027 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh"] Nov 02 22:39:14.490021 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:39:14.489993 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ee8e51_3153_4acc_b025_c964a52be17d.slice/crio-aa028a8525d645d671c7b73659a31d01d6af32af57bdf39c9d6b2819fb13c2ae WatchSource:0}: Error finding container aa028a8525d645d671c7b73659a31d01d6af32af57bdf39c9d6b2819fb13c2ae: Status 404 returned error can't find the container with id aa028a8525d645d671c7b73659a31d01d6af32af57bdf39c9d6b2819fb13c2ae Nov 02 22:39:15.172273 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:15.172236 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" event={"ID":"72ee8e51-3153-4acc-b025-c964a52be17d","Type":"ContainerStarted","Data":"fcc1cdd7991cbae30989cc2e06720ded90bc9efb50fefbb707eb0c3342bad336"} Nov 02 22:39:15.172484 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:15.172292 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" event={"ID":"72ee8e51-3153-4acc-b025-c964a52be17d","Type":"ContainerStarted","Data":"aa028a8525d645d671c7b73659a31d01d6af32af57bdf39c9d6b2819fb13c2ae"} Nov 02 22:39:15.172567 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:15.172503 4309 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" secret="" err="secret \"maas-api-cleanup-dockercfg-n7t66\" not found" Nov 02 22:39:15.188630 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:15.188580 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" podStartSLOduration=2.188567504 podStartE2EDuration="2.188567504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:39:15.186825738 +0000 UTC m=+2483.124938956" watchObservedRunningTime="2026-11-02 22:39:15.188567504 +0000 UTC m=+2483.126680722" Nov 02 22:39:16.179517 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:16.179477 4309 generic.go:356] "Generic (PLEG): container finished" podID="72ee8e51-3153-4acc-b025-c964a52be17d" containerID="fcc1cdd7991cbae30989cc2e06720ded90bc9efb50fefbb707eb0c3342bad336" exitCode=0 Nov 02 22:39:16.179967 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:16.179567 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" event={"ID":"72ee8e51-3153-4acc-b025-c964a52be17d","Type":"ContainerDied","Data":"fcc1cdd7991cbae30989cc2e06720ded90bc9efb50fefbb707eb0c3342bad336"} Nov 02 22:39:17.323158 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.323129 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:17.456268 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.456181 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-openshift-ca\") pod \"72ee8e51-3153-4acc-b025-c964a52be17d\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " Nov 02 22:39:17.456268 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.456243 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj\" (UniqueName: \"kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj\") pod \"72ee8e51-3153-4acc-b025-c964a52be17d\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " Nov 02 22:39:17.456524 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.456275 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-xks-ca\") pod \"72ee8e51-3153-4acc-b025-c964a52be17d\" (UID: \"72ee8e51-3153-4acc-b025-c964a52be17d\") " Nov 02 22:39:17.456586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.456518 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-openshift-ca" pod "72ee8e51-3153-4acc-b025-c964a52be17d" (UID: "72ee8e51-3153-4acc-b025-c964a52be17d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:39:17.458451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.458427 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-xks-ca" pod "72ee8e51-3153-4acc-b025-c964a52be17d" (UID: "72ee8e51-3153-4acc-b025-c964a52be17d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:39:17.458451 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.458440 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj" pod "72ee8e51-3153-4acc-b025-c964a52be17d" (UID: "72ee8e51-3153-4acc-b025-c964a52be17d"). InnerVolumeSpecName "kube-api-access-fq4hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:39:17.557206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.557165 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4hj\" (UniqueName: \"kubernetes.io/projected/72ee8e51-3153-4acc-b025-c964a52be17d-kube-api-access-fq4hj\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:39:17.557206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.557201 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:39:17.557206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:17.557214 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ee8e51-3153-4acc-b025-c964a52be17d-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:39:18.192118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:18.192089 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" Nov 02 22:39:18.192118 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:18.192106 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh" event={"ID":"72ee8e51-3153-4acc-b025-c964a52be17d","Type":"ContainerDied","Data":"aa028a8525d645d671c7b73659a31d01d6af32af57bdf39c9d6b2819fb13c2ae"} Nov 02 22:39:18.192333 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:18.192157 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa028a8525d645d671c7b73659a31d01d6af32af57bdf39c9d6b2819fb13c2ae" Nov 02 22:39:24.265989 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:24.265952 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh"] Nov 02 22:39:24.268325 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:24.268300 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-555ba599-qh6fh"] Nov 02 22:39:26.188206 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:39:26.188171 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ee8e51-3153-4acc-b025-c964a52be17d" path="/var/lib/kubelet/pods/72ee8e51-3153-4acc-b025-c964a52be17d/volumes" Nov 02 22:39:41.594905 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:39:41.594868 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:40:01.385548 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:40:01.385518 4309 scope.go:122] "RemoveContainer" containerID="97d6633a9ae6922b3a332a6f6ea85528c94c91432f8691a265ad2711412d7c36" Nov 02 22:40:01.394472 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:40:01.394449 4309 scope.go:122] "RemoveContainer" containerID="7daedc140c31b2f436f40710ab52d8cdcaee8daa48f92bc4c39387f5ca7ea520" Nov 02 22:40:01.402632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:40:01.402611 4309 scope.go:122] "RemoveContainer" containerID="d8510f1c45bc44c73f987620582f321a0e35f29666e8780e1ce9901ef447b7b5" Nov 02 22:40:11.597076 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:40:11.597037 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:40:34.020488 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:40:34.020456 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8"] Nov 02 22:40:34.024152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:40:34.024125 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-j2rz8"] Nov 02 22:40:34.186807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:40:34.186777 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd72a4fb-b392-40ec-93e5-641eabdc6d80" path="/var/lib/kubelet/pods/fd72a4fb-b392-40ec-93e5-641eabdc6d80/volumes" Nov 02 22:40:41.595994 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:40:41.595909 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:40:54.020160 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:40:54.020125 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58"] Nov 02 22:40:54.024240 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:40:54.024211 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9cd58"] Nov 02 22:40:54.193827 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:40:54.193791 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe4f25b-6b12-4871-a13b-9bb313a1548e" path="/var/lib/kubelet/pods/dfe4f25b-6b12-4871-a13b-9bb313a1548e/volumes" Nov 02 22:41:09.228444 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.228405 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh"] Nov 02 22:41:09.228953 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.228938 4309 cpu_manager.go:414] "containerMap: removing container" podUID="72ee8e51-3153-4acc-b025-c964a52be17d" containerName="revoke-keys" Nov 02 22:41:09.229004 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.228954 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ee8e51-3153-4acc-b025-c964a52be17d" containerName="revoke-keys" Nov 02 22:41:09.229042 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.229036 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ee8e51-3153-4acc-b025-c964a52be17d" containerName="revoke-keys" Nov 02 22:41:09.232715 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.232695 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.238456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.238414 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:41:09.238709 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.238480 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:41:09.238808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.238572 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g84jk\"" Nov 02 22:41:09.238808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.238613 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:41:09.239701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.239665 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh"] Nov 02 22:41:09.392851 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.392809 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.393055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.392855 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxrj\" (UniqueName: \"kubernetes.io/projected/11823ae1-988c-4ed3-b62d-b53694f762cf-kube-api-access-4bxrj\") pod \"maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.393055 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.392956 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.493757 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.493635 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.493946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.493811 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.493946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.493846 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxrj\" (UniqueName: \"kubernetes.io/projected/11823ae1-988c-4ed3-b62d-b53694f762cf-kube-api-access-4bxrj\") pod \"maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.494395 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.494374 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.496017 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.495994 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.501528 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.501494 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxrj\" (UniqueName: \"kubernetes.io/projected/11823ae1-988c-4ed3-b62d-b53694f762cf-kube-api-access-4bxrj\") pod \"maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.550569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.550538 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:09.825666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:09.825640 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh"] Nov 02 22:41:09.827503 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:41:09.827467 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11823ae1_988c_4ed3_b62d_b53694f762cf.slice/crio-40d672380e8af84071912013b70a89717a04ee725d2777e91c39e55e05510e49 WatchSource:0}: Error finding container 40d672380e8af84071912013b70a89717a04ee725d2777e91c39e55e05510e49: Status 404 returned error can't find the container with id 40d672380e8af84071912013b70a89717a04ee725d2777e91c39e55e05510e49 Nov 02 22:41:10.776096 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:10.776059 4309 generic.go:356] "Generic (PLEG): container finished" podID="11823ae1-988c-4ed3-b62d-b53694f762cf" containerID="2abf8a5ee1ec3cba8d94b1313ace9b70c2728402040311e201b6429221cac014" exitCode=0 Nov 02 22:41:10.776537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:10.776144 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" event={"ID":"11823ae1-988c-4ed3-b62d-b53694f762cf","Type":"ContainerDied","Data":"2abf8a5ee1ec3cba8d94b1313ace9b70c2728402040311e201b6429221cac014"} Nov 02 22:41:10.776537 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:10.776195 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" event={"ID":"11823ae1-988c-4ed3-b62d-b53694f762cf","Type":"ContainerStarted","Data":"40d672380e8af84071912013b70a89717a04ee725d2777e91c39e55e05510e49"} Nov 02 22:41:11.596999 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:41:11.596962 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:41:11.919808 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:11.919781 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:12.021539 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.021494 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-xks-ca\") pod \"11823ae1-988c-4ed3-b62d-b53694f762cf\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " Nov 02 22:41:12.021724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.021546 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11823ae1-988c-4ed3-b62d-b53694f762cf-kube-api-access-4bxrj\" (UniqueName: \"kubernetes.io/projected/11823ae1-988c-4ed3-b62d-b53694f762cf-kube-api-access-4bxrj\") pod \"11823ae1-988c-4ed3-b62d-b53694f762cf\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " Nov 02 22:41:12.021724 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.021592 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-openshift-ca\") pod \"11823ae1-988c-4ed3-b62d-b53694f762cf\" (UID: \"11823ae1-988c-4ed3-b62d-b53694f762cf\") " Nov 02 22:41:12.022105 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.022064 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-openshift-ca" pod "11823ae1-988c-4ed3-b62d-b53694f762cf" (UID: "11823ae1-988c-4ed3-b62d-b53694f762cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:41:12.023734 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.023709 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11823ae1-988c-4ed3-b62d-b53694f762cf-kube-api-access-4bxrj" pod "11823ae1-988c-4ed3-b62d-b53694f762cf" (UID: "11823ae1-988c-4ed3-b62d-b53694f762cf"). InnerVolumeSpecName "kube-api-access-4bxrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:41:12.023857 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.023834 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-xks-ca" pod "11823ae1-988c-4ed3-b62d-b53694f762cf" (UID: "11823ae1-988c-4ed3-b62d-b53694f762cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:41:12.123117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.123086 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:12.123117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.123113 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bxrj\" (UniqueName: \"kubernetes.io/projected/11823ae1-988c-4ed3-b62d-b53694f762cf-kube-api-access-4bxrj\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:12.123301 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.123124 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11823ae1-988c-4ed3-b62d-b53694f762cf-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:12.787413 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.787380 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" Nov 02 22:41:12.787738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.787410 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh" event={"ID":"11823ae1-988c-4ed3-b62d-b53694f762cf","Type":"ContainerDied","Data":"40d672380e8af84071912013b70a89717a04ee725d2777e91c39e55e05510e49"} Nov 02 22:41:12.787738 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:12.787501 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40d672380e8af84071912013b70a89717a04ee725d2777e91c39e55e05510e49" Nov 02 22:41:25.411180 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:25.411143 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh"] Nov 02 22:41:25.413134 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:25.413110 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-30c892-mr8fh"] Nov 02 22:41:26.187317 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:26.187273 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11823ae1-988c-4ed3-b62d-b53694f762cf" path="/var/lib/kubelet/pods/11823ae1-988c-4ed3-b62d-b53694f762cf/volumes" Nov 02 22:41:41.678633 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:41:41.678583 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:41:43.137237 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.137201 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9"] Nov 02 22:41:43.137770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.137751 4309 cpu_manager.go:414] "containerMap: removing container" podUID="11823ae1-988c-4ed3-b62d-b53694f762cf" containerName="revoke-keys" Nov 02 22:41:43.137770 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.137770 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11823ae1-988c-4ed3-b62d-b53694f762cf" containerName="revoke-keys" Nov 02 22:41:43.137887 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.137867 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="11823ae1-988c-4ed3-b62d-b53694f762cf" containerName="revoke-keys" Nov 02 22:41:43.141242 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.141226 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.148313 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.148289 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9"] Nov 02 22:41:43.148509 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.148485 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:41:43.148598 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.148539 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:41:43.163843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.163819 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:41:43.163843 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.163825 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g84jk\"" Nov 02 22:41:43.303639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.303604 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xs9z9\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.303639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.303641 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xs9z9\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.303913 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.303716 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dgf2\" (UniqueName: \"kubernetes.io/projected/98682de9-eb95-453b-8323-e4251dfd1e8b-kube-api-access-4dgf2\") pod \"maas-api-revoke-sub-f03629db-xs9z9\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.405231 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.405146 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgf2\" (UniqueName: \"kubernetes.io/projected/98682de9-eb95-453b-8323-e4251dfd1e8b-kube-api-access-4dgf2\") pod \"maas-api-revoke-sub-f03629db-xs9z9\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.405399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.405244 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xs9z9\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.405399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.405264 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xs9z9\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.405892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.405874 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xs9z9\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.407573 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.407545 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xs9z9\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.413247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.413218 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgf2\" (UniqueName: \"kubernetes.io/projected/98682de9-eb95-453b-8323-e4251dfd1e8b-kube-api-access-4dgf2\") pod \"maas-api-revoke-sub-f03629db-xs9z9\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.455324 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.455289 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:43.728666 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.728646 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9"] Nov 02 22:41:43.730417 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:41:43.730389 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98682de9_eb95_453b_8323_e4251dfd1e8b.slice/crio-d1052abca2a5376cd33f609e00f7513a7099d556601b97a3a6b6cf49f85e88e6 WatchSource:0}: Error finding container d1052abca2a5376cd33f609e00f7513a7099d556601b97a3a6b6cf49f85e88e6: Status 404 returned error can't find the container with id d1052abca2a5376cd33f609e00f7513a7099d556601b97a3a6b6cf49f85e88e6 Nov 02 22:41:43.991780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:43.991744 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" event={"ID":"98682de9-eb95-453b-8323-e4251dfd1e8b","Type":"ContainerStarted","Data":"d1052abca2a5376cd33f609e00f7513a7099d556601b97a3a6b6cf49f85e88e6"} Nov 02 22:41:44.998095 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:44.998061 4309 generic.go:356] "Generic (PLEG): container finished" podID="98682de9-eb95-453b-8323-e4251dfd1e8b" containerID="6e7e5d01f26d2f3ff21e5558edd712b34e6c88a8ab84dc9b3b9354181e1110a8" exitCode=0 Nov 02 22:41:44.998482 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:44.998102 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" event={"ID":"98682de9-eb95-453b-8323-e4251dfd1e8b","Type":"ContainerDied","Data":"6e7e5d01f26d2f3ff21e5558edd712b34e6c88a8ab84dc9b3b9354181e1110a8"} Nov 02 22:41:46.134384 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.134360 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:46.235394 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.235360 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-openshift-ca\") pod \"98682de9-eb95-453b-8323-e4251dfd1e8b\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " Nov 02 22:41:46.235590 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.235531 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-xks-ca\") pod \"98682de9-eb95-453b-8323-e4251dfd1e8b\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " Nov 02 22:41:46.235647 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.235603 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98682de9-eb95-453b-8323-e4251dfd1e8b-kube-api-access-4dgf2\" (UniqueName: \"kubernetes.io/projected/98682de9-eb95-453b-8323-e4251dfd1e8b-kube-api-access-4dgf2\") pod \"98682de9-eb95-453b-8323-e4251dfd1e8b\" (UID: \"98682de9-eb95-453b-8323-e4251dfd1e8b\") " Nov 02 22:41:46.235848 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.235826 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-openshift-ca" pod "98682de9-eb95-453b-8323-e4251dfd1e8b" (UID: "98682de9-eb95-453b-8323-e4251dfd1e8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:41:46.236043 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.236025 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:46.237621 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.237592 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98682de9-eb95-453b-8323-e4251dfd1e8b-kube-api-access-4dgf2" pod "98682de9-eb95-453b-8323-e4251dfd1e8b" (UID: "98682de9-eb95-453b-8323-e4251dfd1e8b"). InnerVolumeSpecName "kube-api-access-4dgf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:41:46.237752 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.237706 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-xks-ca" pod "98682de9-eb95-453b-8323-e4251dfd1e8b" (UID: "98682de9-eb95-453b-8323-e4251dfd1e8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:41:46.336690 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.336647 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98682de9-eb95-453b-8323-e4251dfd1e8b-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:46.336835 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:46.336701 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dgf2\" (UniqueName: \"kubernetes.io/projected/98682de9-eb95-453b-8323-e4251dfd1e8b-kube-api-access-4dgf2\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:41:47.009319 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:47.009289 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" Nov 02 22:41:47.009319 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:47.009308 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xs9z9" event={"ID":"98682de9-eb95-453b-8323-e4251dfd1e8b","Type":"ContainerDied","Data":"d1052abca2a5376cd33f609e00f7513a7099d556601b97a3a6b6cf49f85e88e6"} Nov 02 22:41:47.009557 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:47.009370 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1052abca2a5376cd33f609e00f7513a7099d556601b97a3a6b6cf49f85e88e6" Nov 02 22:41:57.067929 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:57.067898 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm"] Nov 02 22:41:57.081094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:57.081038 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d4115f8-bp7xm"] Nov 02 22:41:58.226712 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:58.213540 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f5b07e-61d6-4657-876c-3773c5634c63" path="/var/lib/kubelet/pods/b6f5b07e-61d6-4657-876c-3773c5634c63/volumes" Nov 02 22:41:59.016910 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.016867 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh"] Nov 02 22:41:59.017327 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.017314 4309 cpu_manager.go:414] "containerMap: removing container" podUID="98682de9-eb95-453b-8323-e4251dfd1e8b" containerName="revoke-keys" Nov 02 22:41:59.017375 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.017328 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98682de9-eb95-453b-8323-e4251dfd1e8b" containerName="revoke-keys" Nov 02 22:41:59.017414 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.017400 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="98682de9-eb95-453b-8323-e4251dfd1e8b" containerName="revoke-keys" Nov 02 22:41:59.020762 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.020745 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.028962 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.028645 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:41:59.028962 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.028801 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g84jk\"" Nov 02 22:41:59.028962 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.028857 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:41:59.028962 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.028913 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:41:59.031098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.031074 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh"] Nov 02 22:41:59.096855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.096822 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfwsh\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.096855 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.096858 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfwsh\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.097100 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.096882 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ft7n\" (UniqueName: \"kubernetes.io/projected/da1030e9-3e91-4889-a6cc-3607d0396697-kube-api-access-7ft7n\") pod \"maas-api-revoke-sub-0165d7d2-hfwsh\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.198111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.198079 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfwsh\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.198111 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.198113 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfwsh\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.198372 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.198146 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ft7n\" (UniqueName: \"kubernetes.io/projected/da1030e9-3e91-4889-a6cc-3607d0396697-kube-api-access-7ft7n\") pod \"maas-api-revoke-sub-0165d7d2-hfwsh\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.198847 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.198824 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfwsh\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.200573 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.200547 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hfwsh\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.205876 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.205857 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ft7n\" (UniqueName: \"kubernetes.io/projected/da1030e9-3e91-4889-a6cc-3607d0396697-kube-api-access-7ft7n\") pod \"maas-api-revoke-sub-0165d7d2-hfwsh\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.335132 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.335103 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:41:59.607526 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:41:59.607483 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh"] Nov 02 22:41:59.610713 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:41:59.610656 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1030e9_3e91_4889_a6cc_3607d0396697.slice/crio-08fe6497a76718500f4baf47315710cfdad66cb41fbc6726e545d0c90aa95bc6 WatchSource:0}: Error finding container 08fe6497a76718500f4baf47315710cfdad66cb41fbc6726e545d0c90aa95bc6: Status 404 returned error can't find the container with id 08fe6497a76718500f4baf47315710cfdad66cb41fbc6726e545d0c90aa95bc6 Nov 02 22:42:00.090551 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:00.090512 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" event={"ID":"da1030e9-3e91-4889-a6cc-3607d0396697","Type":"ContainerStarted","Data":"3e36d8cad18109bac892d2ec661f5652f9f717e3198bcf39b0860e24721bd64a"} Nov 02 22:42:00.090739 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:00.090598 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" event={"ID":"da1030e9-3e91-4889-a6cc-3607d0396697","Type":"ContainerStarted","Data":"08fe6497a76718500f4baf47315710cfdad66cb41fbc6726e545d0c90aa95bc6"} Nov 02 22:42:00.106892 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:00.106846 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" podStartSLOduration=2.106833256 podStartE2EDuration="2.106833256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:42:00.105259459 +0000 UTC m=+2648.043372678" watchObservedRunningTime="2026-11-02 22:42:00.106833256 +0000 UTC m=+2648.044946475" Nov 02 22:42:01.098286 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:01.098251 4309 generic.go:356] "Generic (PLEG): container finished" podID="da1030e9-3e91-4889-a6cc-3607d0396697" containerID="3e36d8cad18109bac892d2ec661f5652f9f717e3198bcf39b0860e24721bd64a" exitCode=0 Nov 02 22:42:01.098669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:01.098299 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" event={"ID":"da1030e9-3e91-4889-a6cc-3607d0396697","Type":"ContainerDied","Data":"3e36d8cad18109bac892d2ec661f5652f9f717e3198bcf39b0860e24721bd64a"} Nov 02 22:42:01.440586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:01.440507 4309 scope.go:122] "RemoveContainer" containerID="a73626041e17748e028c0749eabe0a51e1dc710cefa91c34c8cd53bd31f56991" Nov 02 22:42:01.449586 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:01.449567 4309 scope.go:122] "RemoveContainer" containerID="16820c38faf34f0e26d0fcff7caeb0165bc8a86779a4a6308c81cd0a69c9c033" Nov 02 22:42:02.237951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.237925 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:42:02.327970 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.327937 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1030e9-3e91-4889-a6cc-3607d0396697-kube-api-access-7ft7n\" (UniqueName: \"kubernetes.io/projected/da1030e9-3e91-4889-a6cc-3607d0396697-kube-api-access-7ft7n\") pod \"da1030e9-3e91-4889-a6cc-3607d0396697\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " Nov 02 22:42:02.328147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.327996 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-xks-ca\") pod \"da1030e9-3e91-4889-a6cc-3607d0396697\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " Nov 02 22:42:02.328147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.328070 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-openshift-ca\") pod \"da1030e9-3e91-4889-a6cc-3607d0396697\" (UID: \"da1030e9-3e91-4889-a6cc-3607d0396697\") " Nov 02 22:42:02.328426 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.328397 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-openshift-ca" pod "da1030e9-3e91-4889-a6cc-3607d0396697" (UID: "da1030e9-3e91-4889-a6cc-3607d0396697"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:42:02.328569 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.328524 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:02.330138 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.330119 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-xks-ca" pod "da1030e9-3e91-4889-a6cc-3607d0396697" (UID: "da1030e9-3e91-4889-a6cc-3607d0396697"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:42:02.330230 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.330215 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1030e9-3e91-4889-a6cc-3607d0396697-kube-api-access-7ft7n" pod "da1030e9-3e91-4889-a6cc-3607d0396697" (UID: "da1030e9-3e91-4889-a6cc-3607d0396697"). InnerVolumeSpecName "kube-api-access-7ft7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:42:02.429165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.429073 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ft7n\" (UniqueName: \"kubernetes.io/projected/da1030e9-3e91-4889-a6cc-3607d0396697-kube-api-access-7ft7n\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:02.429165 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:02.429108 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1030e9-3e91-4889-a6cc-3607d0396697-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:03.109901 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:03.109871 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" Nov 02 22:42:03.110057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:03.109865 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hfwsh" event={"ID":"da1030e9-3e91-4889-a6cc-3607d0396697","Type":"ContainerDied","Data":"08fe6497a76718500f4baf47315710cfdad66cb41fbc6726e545d0c90aa95bc6"} Nov 02 22:42:03.110057 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:03.110000 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08fe6497a76718500f4baf47315710cfdad66cb41fbc6726e545d0c90aa95bc6" Nov 02 22:42:09.780088 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.779989 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8"] Nov 02 22:42:09.780565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.780465 4309 cpu_manager.go:414] "containerMap: removing container" podUID="da1030e9-3e91-4889-a6cc-3607d0396697" containerName="revoke-keys" Nov 02 22:42:09.780565 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.780478 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1030e9-3e91-4889-a6cc-3607d0396697" containerName="revoke-keys" Nov 02 22:42:09.780716 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.780583 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1030e9-3e91-4889-a6cc-3607d0396697" containerName="revoke-keys" Nov 02 22:42:09.783987 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.783966 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:09.789077 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.789054 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 22:42:09.789359 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.789336 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 22:42:09.789978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.789934 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g84jk\"" Nov 02 22:42:09.790251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.790236 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 22:42:09.792505 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.792478 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8"] Nov 02 22:42:09.903866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.903835 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-z4mq8\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:09.903866 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.903873 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n24nh\" (UniqueName: \"kubernetes.io/projected/71dae70c-d64f-4e23-9401-0b3265cf7aae-kube-api-access-n24nh\") pod \"maas-api-revoke-sub-6b19338f-z4mq8\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:09.904104 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:09.903900 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-z4mq8\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:10.004875 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:10.004832 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-z4mq8\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:10.005092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:10.004885 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n24nh\" (UniqueName: \"kubernetes.io/projected/71dae70c-d64f-4e23-9401-0b3265cf7aae-kube-api-access-n24nh\") pod \"maas-api-revoke-sub-6b19338f-z4mq8\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:10.005092 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:10.004945 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-z4mq8\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:10.005669 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:10.005642 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-z4mq8\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:10.007176 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:10.007155 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-z4mq8\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:10.012632 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:10.012607 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n24nh\" (UniqueName: \"kubernetes.io/projected/71dae70c-d64f-4e23-9401-0b3265cf7aae-kube-api-access-n24nh\") pod \"maas-api-revoke-sub-6b19338f-z4mq8\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:10.104568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:10.104533 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:10.387192 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:10.387154 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8"] Nov 02 22:42:10.389931 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:42:10.389897 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dae70c_d64f_4e23_9401_0b3265cf7aae.slice/crio-9a38a9b73be00c6c6dc3331f1048343faf866681c70f5bacfd02d3987005ffe5 WatchSource:0}: Error finding container 9a38a9b73be00c6c6dc3331f1048343faf866681c70f5bacfd02d3987005ffe5: Status 404 returned error can't find the container with id 9a38a9b73be00c6c6dc3331f1048343faf866681c70f5bacfd02d3987005ffe5 Nov 02 22:42:11.151613 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:11.151575 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" event={"ID":"71dae70c-d64f-4e23-9401-0b3265cf7aae","Type":"ContainerStarted","Data":"c1550800501432dae01e2017691e8f923d7ba2ec5ca30413522de401cf749538"} Nov 02 22:42:11.152018 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:11.151633 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" event={"ID":"71dae70c-d64f-4e23-9401-0b3265cf7aae","Type":"ContainerStarted","Data":"9a38a9b73be00c6c6dc3331f1048343faf866681c70f5bacfd02d3987005ffe5"} Nov 02 22:42:11.168780 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:11.168726 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" podStartSLOduration=2.1687124349999998 podStartE2EDuration="2.168712435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 22:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:42:11.165953812 +0000 UTC m=+2659.104067099" watchObservedRunningTime="2026-11-02 22:42:11.168712435 +0000 UTC m=+2659.106825652" Nov 02 22:42:11.596480 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:42:11.596442 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:42:12.158991 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:12.158954 4309 generic.go:356] "Generic (PLEG): container finished" podID="71dae70c-d64f-4e23-9401-0b3265cf7aae" containerID="c1550800501432dae01e2017691e8f923d7ba2ec5ca30413522de401cf749538" exitCode=0 Nov 02 22:42:12.159405 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:12.159042 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" event={"ID":"71dae70c-d64f-4e23-9401-0b3265cf7aae","Type":"ContainerDied","Data":"c1550800501432dae01e2017691e8f923d7ba2ec5ca30413522de401cf749538"} Nov 02 22:42:13.297418 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.297397 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:13.441631 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.441543 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-xks-ca\") pod \"71dae70c-d64f-4e23-9401-0b3265cf7aae\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " Nov 02 22:42:13.441831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.441651 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-openshift-ca\") pod \"71dae70c-d64f-4e23-9401-0b3265cf7aae\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " Nov 02 22:42:13.441831 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.441706 4309 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71dae70c-d64f-4e23-9401-0b3265cf7aae-kube-api-access-n24nh\" (UniqueName: \"kubernetes.io/projected/71dae70c-d64f-4e23-9401-0b3265cf7aae-kube-api-access-n24nh\") pod \"71dae70c-d64f-4e23-9401-0b3265cf7aae\" (UID: \"71dae70c-d64f-4e23-9401-0b3265cf7aae\") " Nov 02 22:42:13.442020 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.441996 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-openshift-ca" pod "71dae70c-d64f-4e23-9401-0b3265cf7aae" (UID: "71dae70c-d64f-4e23-9401-0b3265cf7aae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 22:42:13.443954 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.443918 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-xks-ca" pod "71dae70c-d64f-4e23-9401-0b3265cf7aae" (UID: "71dae70c-d64f-4e23-9401-0b3265cf7aae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 22:42:13.444093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.443968 4309 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dae70c-d64f-4e23-9401-0b3265cf7aae-kube-api-access-n24nh" pod "71dae70c-d64f-4e23-9401-0b3265cf7aae" (UID: "71dae70c-d64f-4e23-9401-0b3265cf7aae"). InnerVolumeSpecName "kube-api-access-n24nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 22:42:13.543467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.543431 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-openshift-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:13.543467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.543462 4309 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n24nh\" (UniqueName: \"kubernetes.io/projected/71dae70c-d64f-4e23-9401-0b3265cf7aae-kube-api-access-n24nh\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:13.543467 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:13.543474 4309 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dae70c-d64f-4e23-9401-0b3265cf7aae-maas-api-xks-ca\") on node \"ip-10-0-136-9.us-east-2.compute.internal\" DevicePath \"\"" Nov 02 22:42:14.170575 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:14.170538 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" event={"ID":"71dae70c-d64f-4e23-9401-0b3265cf7aae","Type":"ContainerDied","Data":"9a38a9b73be00c6c6dc3331f1048343faf866681c70f5bacfd02d3987005ffe5"} Nov 02 22:42:14.170753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:14.170585 4309 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a38a9b73be00c6c6dc3331f1048343faf866681c70f5bacfd02d3987005ffe5" Nov 02 22:42:14.170753 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:14.170546 4309 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-z4mq8" Nov 02 22:42:31.213531 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:31.213491 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d64dfbf7-vm8gr_b7f87410-f023-4314-b687-5fac8d1922f4/manager/0.log" Nov 02 22:42:33.701360 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:33.701328 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hfwsh_da1030e9-3e91-4889-a6cc-3607d0396697/revoke-keys/0.log" Nov 02 22:42:33.871458 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:33.871431 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0da9d302-p2lvl_78cbfcb2-5a29-4cb8-a648-aefc752ef61d/revoke-keys/0.log" Nov 02 22:42:34.044093 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:34.044019 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-9mzsg_3bfb2ad2-b822-494c-ab26-7766b233ab26/revoke-keys/0.log" Nov 02 22:42:34.205207 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:34.205176 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-6v4d7_0bf06653-f7ea-406a-86f2-c41aa9f59d28/revoke-keys/0.log" Nov 02 22:42:34.381801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:34.381773 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-z4mq8_71dae70c-d64f-4e23-9401-0b3265cf7aae/revoke-keys/0.log" Nov 02 22:42:34.568053 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:34.568025 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xs9z9_98682de9-eb95-453b-8323-e4251dfd1e8b/revoke-keys/0.log" Nov 02 22:42:36.752345 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:36.752318 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-97ts7_21e8f035-97b3-4a9e-b0c0-e91946d9d92d/kuadrant-console-plugin/0.log" Nov 02 22:42:36.974948 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:36.974911 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hwzg2_aa0e89fc-92fe-4b69-9f63-6e94c05a01e0/manager/0.log" Nov 02 22:42:38.513838 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:38.513798 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747dfd5df-bxjlq_e7d8a26d-d0db-4eb3-b5de-75a070848241/kube-auth-proxy/0.log" Nov 02 22:42:39.589996 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:39.589958 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-r8nbz_1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2/istio-proxy/0.log" Nov 02 22:42:39.753200 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:39.753166 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-skqp6_f53ed47c-5e7c-4080-9452-63f49808674e/istio-proxy/0.log" Nov 02 22:42:40.275124 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:40.275096 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-xlsg6_e9c4918d-e8b3-4e8a-a8ee-651b35456661/istio-proxy/0.log" Nov 02 22:42:40.826966 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:40.826932 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qhdt6_20ac817a-ffba-42a8-b379-1fa39f0f81da/payload-pre-processing/0.log" Nov 02 22:42:41.599270 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:42:41.599228 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:42:42.251135 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:42.251099 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-df7vx_d2b2a428-093b-4d9c-8b88-ad2b24b447c8/main/0.log" Nov 02 22:42:42.428639 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:42.428608 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xgv2l_18e45205-8cd3-40c2-b688-6d31bc145c0a/main/0.log" Nov 02 22:42:42.957203 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:42.957170 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hzbck_774397e4-c8ee-4734-8f5a-7befd9b4d9dc/main/0.log" Nov 02 22:42:43.124801 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:43.124772 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl_96e4208b-1bbe-4a26-a5c0-77783d8239b9/main/0.log" Nov 02 22:42:53.127568 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:53.127530 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d64dfbf7-vm8gr_b7f87410-f023-4314-b687-5fac8d1922f4/manager/0.log" Nov 02 22:42:55.448319 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:55.448289 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-hfwsh_da1030e9-3e91-4889-a6cc-3607d0396697/revoke-keys/0.log" Nov 02 22:42:55.613353 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:55.613324 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0da9d302-p2lvl_78cbfcb2-5a29-4cb8-a648-aefc752ef61d/revoke-keys/0.log" Nov 02 22:42:55.778355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:55.778273 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-9mzsg_3bfb2ad2-b822-494c-ab26-7766b233ab26/revoke-keys/0.log" Nov 02 22:42:55.948215 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:55.948181 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-6v4d7_0bf06653-f7ea-406a-86f2-c41aa9f59d28/revoke-keys/0.log" Nov 02 22:42:56.128611 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:56.128582 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-z4mq8_71dae70c-d64f-4e23-9401-0b3265cf7aae/revoke-keys/0.log" Nov 02 22:42:56.303452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:56.303425 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-xs9z9_98682de9-eb95-453b-8323-e4251dfd1e8b/revoke-keys/0.log" Nov 02 22:42:56.634117 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:56.634086 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:42:56.645525 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:56.645496 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pgrkb_89a526ea-68b3-40e3-a474-02fa8cec0ab0/ovn-acl-logging/0.log" Nov 02 22:42:58.154113 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:58.154088 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-97ts7_21e8f035-97b3-4a9e-b0c0-e91946d9d92d/kuadrant-console-plugin/0.log" Nov 02 22:42:58.342701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:58.342644 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hwzg2_aa0e89fc-92fe-4b69-9f63-6e94c05a01e0/manager/0.log" Nov 02 22:42:59.599456 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:42:59.599425 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747dfd5df-bxjlq_e7d8a26d-d0db-4eb3-b5de-75a070848241/kube-auth-proxy/0.log" Nov 02 22:43:00.712740 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:00.712712 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-r8nbz_1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2/istio-proxy/0.log" Nov 02 22:43:00.898424 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:00.898391 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-skqp6_f53ed47c-5e7c-4080-9452-63f49808674e/istio-proxy/0.log" Nov 02 22:43:01.422917 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:01.422882 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-xlsg6_e9c4918d-e8b3-4e8a-a8ee-651b35456661/istio-proxy/0.log" Nov 02 22:43:01.475147 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:01.475118 4309 scope.go:122] "RemoveContainer" containerID="89ef9168bb00e1b113472dfbc3de5ceb54a836d71f5c3ca624a869b0741a8661" Nov 02 22:43:01.968807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:01.968776 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qhdt6_20ac817a-ffba-42a8-b379-1fa39f0f81da/payload-pre-processing/0.log" Nov 02 22:43:02.989842 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:02.989812 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-df7vx_d2b2a428-093b-4d9c-8b88-ad2b24b447c8/main/0.log" Nov 02 22:43:03.021188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:03.021153 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7"] Nov 02 22:43:03.023399 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:03.023367 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-6v4d7"] Nov 02 22:43:03.162807 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:03.162778 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xgv2l_18e45205-8cd3-40c2-b688-6d31bc145c0a/main/0.log" Nov 02 22:43:03.705946 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:03.705920 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hzbck_774397e4-c8ee-4734-8f5a-7befd9b4d9dc/main/0.log" Nov 02 22:43:03.897886 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:03.897857 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-c8szl_96e4208b-1bbe-4a26-a5c0-77783d8239b9/main/0.log" Nov 02 22:43:04.186864 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:04.186832 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf06653-f7ea-406a-86f2-c41aa9f59d28" path="/var/lib/kubelet/pods/0bf06653-f7ea-406a-86f2-c41aa9f59d28/volumes" Nov 02 22:43:11.595636 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:43:11.595599 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:43:18.018999 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:18.018966 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg"] Nov 02 22:43:18.023429 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:18.023407 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-9mzsg"] Nov 02 22:43:18.186841 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:18.186811 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bfb2ad2-b822-494c-ab26-7766b233ab26" path="/var/lib/kubelet/pods/3bfb2ad2-b822-494c-ab26-7766b233ab26/volumes" Nov 02 22:43:39.019275 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:39.019192 4309 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl"] Nov 02 22:43:39.021287 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:39.021256 4309 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da9d302-p2lvl"] Nov 02 22:43:40.187273 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:43:40.187239 4309 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cbfcb2-5a29-4cb8-a648-aefc752ef61d" path="/var/lib/kubelet/pods/78cbfcb2-5a29-4cb8-a648-aefc752ef61d/volumes" Nov 02 22:43:41.594606 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:43:41.594563 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:44:01.494994 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:01.494962 4309 scope.go:122] "RemoveContainer" containerID="c241d481813f57958b3e81d25199cabaaacee8983b71500688f032a57af24e24" Nov 02 22:44:01.504446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:01.504421 4309 scope.go:122] "RemoveContainer" containerID="71cbb6976151adea753d154e1cc953256d4e462becbc86bb8131bf50d91d5ff0" Nov 02 22:44:11.594890 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:44:11.594849 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:44:30.113773 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:30.113728 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qkl72_3ee5c83e-ba60-41ef-a638-a77cdff3c824/global-pull-secret-syncer/0.log" Nov 02 22:44:30.356355 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:30.356321 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zxdmr_10158047-423e-4241-b46d-6ad29dece441/konnectivity-agent/0.log" Nov 02 22:44:30.406622 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:30.406548 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-9.us-east-2.compute.internal_ad3b40d52cd903e26b09603b8260563a/haproxy/0.log" Nov 02 22:44:37.578827 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:37.578801 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-97ts7_21e8f035-97b3-4a9e-b0c0-e91946d9d92d/kuadrant-console-plugin/0.log" Nov 02 22:44:37.645002 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:37.644971 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-hwzg2_aa0e89fc-92fe-4b69-9f63-6e94c05a01e0/manager/0.log" Nov 02 22:44:39.007978 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.007945 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1928586d-c241-4777-ba20-d31b032dec7e/alertmanager/0.log" Nov 02 22:44:39.030818 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.030793 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1928586d-c241-4777-ba20-d31b032dec7e/config-reloader/0.log" Nov 02 22:44:39.050779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.050752 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1928586d-c241-4777-ba20-d31b032dec7e/kube-rbac-proxy-web/0.log" Nov 02 22:44:39.070422 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.070398 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1928586d-c241-4777-ba20-d31b032dec7e/kube-rbac-proxy/0.log" Nov 02 22:44:39.093972 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.093950 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1928586d-c241-4777-ba20-d31b032dec7e/kube-rbac-proxy-metric/0.log" Nov 02 22:44:39.113452 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.113430 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1928586d-c241-4777-ba20-d31b032dec7e/prom-label-proxy/0.log" Nov 02 22:44:39.134017 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.133995 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1928586d-c241-4777-ba20-d31b032dec7e/init-config-reloader/0.log" Nov 02 22:44:39.223257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.223224 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k6wjv_0a657025-9495-432b-a966-db8b41cae3df/kube-state-metrics/0.log" Nov 02 22:44:39.238619 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.238594 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k6wjv_0a657025-9495-432b-a966-db8b41cae3df/kube-rbac-proxy-main/0.log" Nov 02 22:44:39.259492 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.259426 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k6wjv_0a657025-9495-432b-a966-db8b41cae3df/kube-rbac-proxy-self/0.log" Nov 02 22:44:39.283434 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.283409 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-79998fd94f-dxckg_9efe0f4f-aa61-43a1-8c50-d2380a6fa2f3/metrics-server/0.log" Nov 02 22:44:39.337644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.337619 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bzm9b_9b08273d-5f57-44e7-896d-61ee487fad84/node-exporter/0.log" Nov 02 22:44:39.357420 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.357397 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bzm9b_9b08273d-5f57-44e7-896d-61ee487fad84/kube-rbac-proxy/0.log" Nov 02 22:44:39.379651 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.379628 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bzm9b_9b08273d-5f57-44e7-896d-61ee487fad84/init-textfile/0.log" Nov 02 22:44:39.640251 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.640221 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b29d959-15eb-45fc-b165-cebbdc693076/prometheus/0.log" Nov 02 22:44:39.659878 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.659848 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b29d959-15eb-45fc-b165-cebbdc693076/config-reloader/0.log" Nov 02 22:44:39.680544 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.680517 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b29d959-15eb-45fc-b165-cebbdc693076/thanos-sidecar/0.log" Nov 02 22:44:39.704197 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.704172 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b29d959-15eb-45fc-b165-cebbdc693076/kube-rbac-proxy-web/0.log" Nov 02 22:44:39.725201 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.725184 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b29d959-15eb-45fc-b165-cebbdc693076/kube-rbac-proxy/0.log" Nov 02 22:44:39.749446 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.749425 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b29d959-15eb-45fc-b165-cebbdc693076/kube-rbac-proxy-thanos/0.log" Nov 02 22:44:39.770983 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.770963 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b29d959-15eb-45fc-b165-cebbdc693076/init-config-reloader/0.log" Nov 02 22:44:39.801779 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.801749 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rt7kj_32361f80-c746-4c97-9e73-20798d0c5417/prometheus-operator/0.log" Nov 02 22:44:39.818670 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.818652 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rt7kj_32361f80-c746-4c97-9e73-20798d0c5417/kube-rbac-proxy/0.log" Nov 02 22:44:39.842245 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.842220 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-26cj5_256801e0-d5c5-4928-8aed-8b86856fc560/prometheus-operator-admission-webhook/0.log" Nov 02 22:44:39.876214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.876190 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dc44596d8-r9jct_6022facf-3f59-4c38-bcb1-d7a8e3853e68/telemeter-client/0.log" Nov 02 22:44:39.902450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.902392 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dc44596d8-r9jct_6022facf-3f59-4c38-bcb1-d7a8e3853e68/reload/0.log" Nov 02 22:44:39.922961 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.922932 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dc44596d8-r9jct_6022facf-3f59-4c38-bcb1-d7a8e3853e68/kube-rbac-proxy/0.log" Nov 02 22:44:39.951001 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.950980 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75fcc597f8-zw9z8_c9ef1d8c-8213-455c-a988-925d8cdaa2b5/thanos-query/0.log" Nov 02 22:44:39.971447 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.971426 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75fcc597f8-zw9z8_c9ef1d8c-8213-455c-a988-925d8cdaa2b5/kube-rbac-proxy-web/0.log" Nov 02 22:44:39.993450 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:39.993424 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75fcc597f8-zw9z8_c9ef1d8c-8213-455c-a988-925d8cdaa2b5/kube-rbac-proxy/0.log" Nov 02 22:44:40.018257 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.018231 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75fcc597f8-zw9z8_c9ef1d8c-8213-455c-a988-925d8cdaa2b5/prom-label-proxy/0.log" Nov 02 22:44:40.042188 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.042165 4309 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg"] Nov 02 22:44:40.042638 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.042625 4309 cpu_manager.go:414] "containerMap: removing container" podUID="71dae70c-d64f-4e23-9401-0b3265cf7aae" containerName="revoke-keys" Nov 02 22:44:40.042701 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.042640 4309 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71dae70c-d64f-4e23-9401-0b3265cf7aae" containerName="revoke-keys" Nov 02 22:44:40.042741 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.042723 4309 memory_manager.go:365] "RemoveStaleState removing state" podUID="71dae70c-d64f-4e23-9401-0b3265cf7aae" containerName="revoke-keys" Nov 02 22:44:40.046204 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.046186 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.055348 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.055327 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg"] Nov 02 22:44:40.057098 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.057079 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75fcc597f8-zw9z8_c9ef1d8c-8213-455c-a988-925d8cdaa2b5/kube-rbac-proxy-rules/0.log" Nov 02 22:44:40.087335 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.087316 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75fcc597f8-zw9z8_c9ef1d8c-8213-455c-a988-925d8cdaa2b5/kube-rbac-proxy-metrics/0.log" Nov 02 22:44:40.143470 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.143442 4309 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7zc4v\"/\"default-dockercfg-pw2c6\"" Nov 02 22:44:40.144903 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.144884 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zc4v\"/\"openshift-service-ca.crt\"" Nov 02 22:44:40.145006 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.144967 4309 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7zc4v\"/\"kube-root-ca.crt\"" Nov 02 22:44:40.197241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.197178 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-podres\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.197241 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.197229 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm88c\" (UniqueName: \"kubernetes.io/projected/98613085-69f9-4951-9ee9-6aca938df076-kube-api-access-jm88c\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.197445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.197343 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-lib-modules\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.197445 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.197400 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-sys\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.197574 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.197456 4309 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-proc\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.298360 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.298325 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm88c\" (UniqueName: \"kubernetes.io/projected/98613085-69f9-4951-9ee9-6aca938df076-kube-api-access-jm88c\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.298536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.298384 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-lib-modules\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.298536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.298430 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-sys\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.298536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.298459 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-proc\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.298536 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.298528 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-sys\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.298782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.298546 4309 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-podres\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.298782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.298528 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-lib-modules\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.298782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.298587 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-proc\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.298782 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.298615 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/98613085-69f9-4951-9ee9-6aca938df076-podres\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.306725 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.306708 4309 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm88c\" (UniqueName: \"kubernetes.io/projected/98613085-69f9-4951-9ee9-6aca938df076-kube-api-access-jm88c\") pod \"perf-node-gather-daemonset-5d8jg\" (UID: \"98613085-69f9-4951-9ee9-6aca938df076\") " pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.359520 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.359492 4309 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.632156 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.632129 4309 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg"] Nov 02 22:44:40.633563 ip-10-0-136-9 kubenswrapper[4309]: W1102 22:44:40.633534 4309 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod98613085_69f9_4951_9ee9_6aca938df076.slice/crio-10651cc77dea5d98f4d74abf549a1d10eb315b0eef7c5ec4e6e2d60962d734bd WatchSource:0}: Error finding container 10651cc77dea5d98f4d74abf549a1d10eb315b0eef7c5ec4e6e2d60962d734bd: Status 404 returned error can't find the container with id 10651cc77dea5d98f4d74abf549a1d10eb315b0eef7c5ec4e6e2d60962d734bd Nov 02 22:44:40.954704 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.954606 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" event={"ID":"98613085-69f9-4951-9ee9-6aca938df076","Type":"ContainerStarted","Data":"0403ba2ab40f142d3e33050a32bdaab2c85045348ca88c2922e52d8446047174"} Nov 02 22:44:40.954863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.954707 4309 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" event={"ID":"98613085-69f9-4951-9ee9-6aca938df076","Type":"ContainerStarted","Data":"10651cc77dea5d98f4d74abf549a1d10eb315b0eef7c5ec4e6e2d60962d734bd"} Nov 02 22:44:40.954863 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.954731 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:40.971934 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:40.971880 4309 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" podStartSLOduration=0.971866809 podStartE2EDuration="971.866809ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 22:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 22:44:40.968915768 +0000 UTC m=+2808.907028987" watchObservedRunningTime="2026-11-02 22:44:40.971866809 +0000 UTC m=+2808.909980026" Nov 02 22:44:41.595111 ip-10-0-136-9 kubenswrapper[4309]: E1102 22:44:41.595070 4309 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hjd95" containerName="extractor" Nov 02 22:44:42.482263 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:42.482229 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-z8gn7_660abcf1-bcc1-486a-a0f1-dec089f5cc42/download-server/0.log" Nov 02 22:44:44.221170 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:44.221146 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qk42s_6894f559-b442-486b-a81d-05e13031bf22/dns/0.log" Nov 02 22:44:44.240408 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:44.240381 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qk42s_6894f559-b442-486b-a81d-05e13031bf22/kube-rbac-proxy/0.log" Nov 02 22:44:44.308900 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:44.308876 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kclzj_735421c1-678c-4cc4-a07e-57ffb0e9ca29/dns-node-resolver/0.log" Nov 02 22:44:44.991019 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:44.990984 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jg7pl_527b2ac7-739e-4f53-b553-474cad1eb712/node-ca/0.log" Nov 02 22:44:46.969749 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:46.969723 4309 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7zc4v/perf-node-gather-daemonset-5d8jg" Nov 02 22:44:49.493964 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:49.493933 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747dfd5df-bxjlq_e7d8a26d-d0db-4eb3-b5de-75a070848241/kube-auth-proxy/0.log" Nov 02 22:44:49.691154 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:49.691123 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-r8nbz_1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2/istio-proxy/0.log" Nov 02 22:44:49.718377 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:49.718351 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-skqp6_f53ed47c-5e7c-4080-9452-63f49808674e/istio-proxy/0.log" Nov 02 22:44:49.807278 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:49.807199 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-xlsg6_e9c4918d-e8b3-4e8a-a8ee-651b35456661/istio-proxy/0.log" Nov 02 22:44:49.888508 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:49.888482 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qhdt6_20ac817a-ffba-42a8-b379-1fa39f0f81da/payload-pre-processing/0.log" Nov 02 22:44:51.764624 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:51.764573 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cgm5k_d2a56b64-0440-4f3a-abc8-fa257a8db420/serve-healthcheck-canary/0.log" Nov 02 22:44:52.328684 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:52.328648 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjd95_c69c738c-eb34-47f9-99b6-5770eab13da4/kube-rbac-proxy/0.log" Nov 02 22:44:52.349247 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:52.349225 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjd95_c69c738c-eb34-47f9-99b6-5770eab13da4/exporter/0.log" Nov 02 22:44:52.370951 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:52.370927 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjd95_c69c738c-eb34-47f9-99b6-5770eab13da4/extractor/0.log" Nov 02 22:44:53.642889 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:53.642856 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5747dfd5df-bxjlq_e7d8a26d-d0db-4eb3-b5de-75a070848241/kube-auth-proxy/0.log" Nov 02 22:44:53.755748 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:53.755715 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-r8nbz_1a56fdcc-fa46-4fba-b2d1-3b7ab8a7c9c2/istio-proxy/0.log" Nov 02 22:44:53.774152 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:53.774123 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-skqp6_f53ed47c-5e7c-4080-9452-63f49808674e/istio-proxy/0.log" Nov 02 22:44:53.828859 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:53.828833 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-696f47c6c-xlsg6_e9c4918d-e8b3-4e8a-a8ee-651b35456661/istio-proxy/0.log" Nov 02 22:44:53.879644 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:53.879614 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qhdt6_20ac817a-ffba-42a8-b379-1fa39f0f81da/payload-pre-processing/0.log" Nov 02 22:44:54.794094 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:54.794066 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c6f6f47c9-6n66w_981b466e-ef40-47ae-89c9-609b5733a75d/manager/0.log" Nov 02 22:44:54.814214 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:54.814192 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-5c6f6f47c9-6n66w_981b466e-ef40-47ae-89c9-609b5733a75d/copy-manifests/0.log" Nov 02 22:44:56.208023 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:56.207993 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-7bd56b5fcb-jnfnh_d091e2d4-a383-49f6-b8fb-4ea544b231b5/manager/0.log" Nov 02 22:44:56.235654 ip-10-0-136-9 kubenswrapper[4309]: I1102 22:44:56.235628 4309 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-l5l2f_315cd700-559d-4511-812c-c55c7d7a2bb8/jobset-operator/0.log"